As a salsa club dancer and a ballroom dancer, I see lots of cleavage in all the venues. In the club I am entertained by what the girls are wearing, and as the night progresses the girls coming in the door are wearing less and less! I don't consider latin ballroom dancers to be conservative either. For example, just take a look at America's Ballroom Challenge, which plays on PBS occassionally and just aired again last week. Both the latin and rhythm dancers, and so many ladies in the audience (some you don't want to see!), are placing their cleavage on display for all to see.
You know there are both classy guys who give a passing glance of appreciation, and classless guys who will stare and be gross. Nothing will ever change this, but I assume that you don't want to cover up because you enjoy the subtle appreciation from the classier guys. So in the end its a matter of weighing the enjoyment against the disgust.