I would have to agree with you there. Remember, there was a bit "anarchist" movement in this country a hundred years ago. All kinds of attacks were happening, a couple of presidents were assassinated, the whole nine. Attitudes toward blacks (& immigrants) probably are what kept the country out of revolution back then.
And actually, I think I read once where unionists were trying to erase racism among white laborers, to unite them against the capitalists. Something like, "they're trying to keep you focused on the black man as your enemy, when you should both be united against them," etc. The unions clearly saw racism as an impediment to success against the corporate powers back in the early 1900s. And undoubtedly, so did the communists, who were pretty active here back then. Deep.