I am Unsuccessful

I have a really hard time telling men from women. In an average day I probably mistake a dozen men for women or vice-versa. (that is, I get confused 24 times.)

It seems like this is something people should just be able to do. Do most other people have this problem?