Some suspected Johnson's burka comments were meant to boost his appeal among right-wing members of the party.
60 per cent - said the comments by the former foreign secretary were not racist, while 33 per cent thought they were.
Speaking on the BBC's World At One, the MP for Beaconsfield said: "If he was to become leader of the party, I won't be in it".
British Prime Minister Theresa May backed calls Tuesday for her former foreign minister Boris Johnson to apologize for disparaging comments he made about Muslim women wearing burqas - but he branded his critics "ridiculous".
The party's code of conduct says that MPs and other holders of public office should "not use their position to bully, abuse, victimise, harass or unlawfully discriminate against others" and "foster respect and tolerance".
In a column in Monday's Daily Telegraph newspaper, Johnson said women wearing the full face veil looked like "bank robbers" or "letter-boxes", prompting accusations of Islamophobia. It promises that complaints will be investigated "in a timely and confidential manner".
"I believe women should be able to choose how they dress", said May.
He added: "This further pushes the boundaries of acceptable political debate into an ever more extreme place".
"These were offensive comments but clever politics", she said. "What starts as useful targets for "colourful political language" and the odd bit of toxic campaigning ends up in attacks on our streets". "He did not commit a criminal offence".