(I am not going to drive away those who have believed.) This alludes to the fact that they (the disbelievers) requested Nuh to disassociate himself from the believers, because they were averse to them and felt themselves too important to sit with them. This is similar to the request of disbelievers to the seal of the Messengers ﷺ to disassociate himself from a group of the people who were considered weak in their social status. They wanted the Prophet to sit with them in a special gathering of the elite. Therefore, Allah revealed,

وَلاَ تَطْرُدِ الَّذِينَ يَدْعُونَ رَبَّهُمْ بِالْغَدَاةِ وَالْعَشِىِّ

(And turn not away those who invoke their Lord, morning and afternoon.) Allah also says,

وَكَذلِكَ فَتَنَّا بَعْضَهُمْ بِبَعْضٍ لِّيَقُولواْ أَهَـؤُلاءِ مَنَّ اللَّهُ عَلَيْهِم مِّن بَيْنِنَآ أَلَيْسَ اللَّهُ بِأَعْلَمَ بِالشَّـكِرِينَ

(Thus We have tried some of them with others, that they might say: "Is it these (poor believers) whom Allah has favored from among us" Does not Allah know best those who are grateful)6:53