I know someone who did that and in the end he went from well-paid technical sales to homeless with serious health issues. One piece of minor bad luck followed by more bad luck with the health issues, probably a consequence of the former. Some of his former colleagues, years later, when one of them had found him in the streets, collected money to help him out.

There is a risk of stepping outside. It's not all roses. People are not afraid for no reason and it's not just "all in the head" and an attitude problem.