While true, this is also the kind of thinking that gets you populist leaders like Donald Trump. You can drive out the opinions of everybody you disagree with, everywhere, while chanting that "you have free speech, just not here!", but as long as we hang on to the concept of representative democracy, those people are still participating in the political process, and they're bound to remember that the ballot box is the only place that they're permitted to express unapproved viewpoints.
How on earth is encouraging people to express views that, followed through, would result in direct harm to people, supposed to prevent those views from being put into policy or resulting in hate crime? The actual problem isn't that they're expressing those views - it's that they have them in the first place. Consider solving that, rather than rushing to the aid of people who'd rather have me dead.
