Being brown in a white place is a different experience than being brown in a brown place. There is a lot of friction when different enough groups coexist. If you've ever found yourself in maybe a job or something where you thought differently enough from the people around you to feel like they were more interested in just maintaining their bubble of how they understand the world than discovering the best way to be, it's that same mechanism in action. Just it's blown up to much larger proportions since it's not just you differing with people who share your race, culture, ideas about how society should function, etc.