No. (but it also depends on where you live whether you may want to take it into account)
If I am by myself and not acting the fool then I expect that people around my are not going to feel uncomfortable.
Quite frankly if a person is that stupid to show discomfort then rest assured I will use it to my advantage to embarrass them for their ignorance.
In the a city where 99% of the crime is committed by whites I find it somewhat offensive to walk by a person who grabs their purse as I approach- so sometimes I make a comment about their stupidity.