Not just by men. My wife and I have ended two friendships with women (one a friend of mine, one a friend of hers) after seeing how our ex-friends treated darker-skinned servers in restaurants.

It is one of those things you can't unsee once you've seen it. Both of us are professionals now, but we both worked (my wife much longer) in service jobs. Seeing our exfriends treating these women like that it is clear to us how they would treat us if we didn't look the way we look (I'm caucasian, my wife is slavic).