On your object detection point, Gemini 2.0 Flash has bounding box detection: https://ai.google.dev/gemini-api/docs/models/gemini-v2#bound....

I haven't found it to work particularly well for some more domain-specific things I tried, but it was surprisingly good for an LLM.