If by attraction you mean physical attraction, I'm going to have to disagree with you as well.
I think there are several kinds of attraction. They can all lead to you wanting to "be with" someone (in the romantic/sexual sense).
-There is PHYSICAL attraction.
-There is EMOTIONAL attraction. This is what you feel with a person you connect to and trust. This person gets your emotions and listens to you. This person trusts you enough to tell you his feelings and be vulnerable around you. He values your opinions. He never discounts or ridicules your feelings, and he has your back when you need it. Trustworthiness and being trusted in return are sexy. Maybe it's a sociobiological instinct: trustworthy guys are more likely to stick around and make better daddies. Or maybe it has nothing to do with that. But it's there.
-There is INTELLECTUAL attraction. Do NOT discount this one! There are people who get your ideas, and you get their ideas. And then the person says "You know, what I think is fascinating about your idea is..." and the person builds on it, in a way that shows how totally he/she gets what you were saying. And that makes you have another good idea. And you realize you are a better person and a smarter person with this other person around. That is a huge turn on! An okay-looking guy with whom one can connect intellectually is sexier to me (and many women like me) than the ultra-hot guy who is dumb as a post. Admittedly, it may take me an hour or two to realize how sexy the first guy is, but I generally don't go to bed with people in the first hour anyway. I'm weird that way.
Yes, there is probably a baseline minimum of physical attractiveness. But I'll bet you would be surprised how low that bar is for many women. I have a friend who is married to a guy with a rather visible physical deformity (missing a few fingers), which I'll bet many people would think would be a disadvantage in the dating game. He is SUPER bright and they share a rather esoteric interest. Attracted to each other? Heck yeah!
***
I should add that one of my somewhat cynical--but very bright (and sexy) friends--proposed the following aphorism:
There are three kinds of attraction: emotional, intellectual, and physical. With any one person, it is usually only possible to have two out of the three."
Not sure if he's right, but I think of this gem from time to time!