Fine-tuned MedPalm is worse than GPT-4 on most Medical Challenge Tests. Fine-tuned Minerva is much worse on arithmetic benchmarks.

The LLM space is just different. There's no guarantee a fine-tuned model will beat a bigger generalist one.