Empirically, this is actually beyond doubt by now. Piketty et al. were in the news 10 years ago with this very topic, showing that inequality is detrimental to growth, for many countries and over very large periods of time. And there has been a bunch of additional studies after that publication that support these findings.