Can women be prejudiced against men? Sure. Can minorities be biased? Yup. But sexism/racism (or any other "ism") isn't only about bias! Rather, "isms" are about bias + power. So no, I'm sorry, biased or not women do not have the same power in society as men nor do minorities have the same power as majorities. Ignoring this profound difference is the luxury of being in a socially privileged position.
Abuse is abuse, racism is racism, and sexism is sexism regardless of the perceived power of the perpetrator relative to that of the victim. And wrong in any case, whether or not the harm is thought to be greater or smaller because of the perceived power imbalance.