English and other languages come with lots of ambiguity and assumptions. A significant benefit of programming languages is they have explicit rules for how they will be converted into a running program. An LLM can take many paths from the same starting prompt and deliver vastly different output.
A nice article on this was in the January issue of the Communications of the ACM [1]. With a reference to a piece by Dijkstra predicting that this is never going to be effective, back in 1979.
Being able to write code in a programming language is a feature, not a flaw. If we had always had to program in natural language, the precision and unambiguity of programming languages would be an eagerly welcomed revolution.
I do agree… perhaps the thing to do is write fragments of the program, like the start and end, asking it to complete the middle. If you have precisely described how the output will be printed, for example, then you have essentially formally specified how the data should be organized…
