> User: Question. LLM: Answer. User: Why A? Why B? LLM: Answer. User: But C. What's the tradeoff? LLM: Answer. User: Couldn't also X? LLM: Answer. User: I'm not familiar with Y, explain.
In practice it’s more like:
User: Question. LLM: Answer. User: Why A? LLM: Actually, you’re right, it’s not A, it’s B. User: Why B? LLM: Actually, you’re right, it’s not B, it’s C. User: Why C? LLM: Actually, you’re right, it’s not C, it’s D.
I’ve had this happen pretty much every time I asked an LLM a non-trivial question.
Change your question asking strategy, and potentially models. GPT-5 is much less prone to do this. That's the easiest change to make, try it out. I'm not saying this to promote OpenAI, have a look at my profile if you must. Qwen is also less likely to do this. Sonnet and GPT4o are obviously famous for this sycophancy, and DeepSeek has clearly been training on Claude outputs, and so is also prone to it.
Of course never use chat models, only straight API, as the system prompts for consumer versions make them induce sycophancy as that is what the uncurious like. For this conversation, don't use things like Claude Code or other harnesses either. Just have a conversation through API calls.
You also changed the questions in the scenario in a way that makes them more likely. Keep them open-ended, the curious' most used phrase should probably be "Why or why not? If so, explain why. If not, explain why not".
In practice it’s more like:
User: Question. LLM: Answer. User: Why A? LLM: Actually, you’re right, it’s not A, it’s B. User: Why B? LLM: Actually, you’re right, it’s not B, it’s C. User: Why C? LLM: Actually, you’re right, it’s not C, it’s D.
I’ve had this happen pretty much every time I asked an LLM a non-trivial question.