not that I disagree with you in principle but it is a bad example. there are a lot more places where rich do not want to live than places where they do so rich outbidding you for housing wouldn't make even a top-100 list of issues. the "If you have enough, why does it matter if someone else has more" is basically "rich get to live in Monaco, you get to live in Elmo, Kansas" with all your needs met
A good location means more opportunities. Someone located in the center of a large expensive city will have a lot more opportunities to make money and meet people who have influence than someone in Elmo. A business set up in Elmo will not make as much money as a business set up in Monaco. This means that the best opportunities are reserved for people who need them the least.
San Francisco was the cheap to live city that allowed poor, non-monetary obsessed youth to move there and live a hippie lifestyle.
Silicon valley was backward cheap farmland that allowed students in the nearby universities to stay in an area with their college friends and start their business ideas instead of moving back home.
Anywhere there is excess energy/synergy the rich move in and try to capture it, sucking it out. You need places where society can grow, where excess energy is allowed to create excitement/progress/try new things.