Women and gays and other out groups get a voice in what qualifies as anti-female, anti-gay, etc. They do not get to decide. Using your logic white supremecists get to decide what is anti-white supremecist. Nazis get to decide what is anti-Nazi, etc.

Neither you nor I nor any single person or group gets to decide unilaterally what is "right" or "the truth" because there is no unilateral "right" on how someone feels. Does a bisexual woman get less right to say what is anti-gay than a lesbian? Who decides who is more gay?

In the end it all ends up with either everyone gets an opinion and society as a whole decides (democracy) or you try to claim that others can't speak because they don't have the right "credentials."

Even the definition of "hate speech" has this challenge.