It can't be done in the LLM itself of course, but the wrapper you're taking about already exists in multiple projects fighting in SWEbench. The simplest one is aider with --auto-test https://aider.chat/docs/usage/lint-test.html

There are also large applications like https://devin.ai/ or https://github.com/AI-App/OpenDevin.OpenDevin