Answer to his though experiment:
Yes, I believe a sufficiently advanced AI could told us that. Scientists who have been fed with wrong information can come up with completely new ideas. Making what we know less wrong.
That being said, I don't think current token-predictors can do that.
My read of this was that AI is fundamentally limited by the lack of access to the new empirical data that drove this discovery; that it couldn't have been inferred from the existing corpus of knowledge.
Recent LLMs have larger context windows to process more data and tool use to get new data, so it would be surprising if there’s a fundamental limitation here.
That being said, I don't think current token-predictors can do that.