IMHO the problem is projection. You have a group that has oppressed minoriities for centuries. This oppresor group can't imagine that the minorities won't become oppressive if the minorities get power / become the majority.

Sadly, we have the glaring example of Israel that supports this fear. Jews have been oppressed, stigmatized and murdered for millennia. They "get their own country" (by taking it from the people already there) and the Israelis are doing to the Palestinians exactly what was done to them, and what white colonists have done to indigenous people for centuries.