> A human being who did that without ever playing a game would also start out better than a typical novice.
I am quite skeptical of these arguments along the lines of “imagine a human read everything written on the topic…”.
What humans are doing when they read something is not what neural nets are doing when they read something. Humans are (idealistically) doing something like Feynman’s description of how he reads (or in this case, listens to) a theorem:
“I had a scheme, which I still use today when somebody is explaining something that I’m trying to understand: I keep making up examples. For instance, the mathematicians would come in with a terrific theorem, and they’re all excited. As they’re telling me the conditions of the theorem, I construct something which fits all the conditions. You know, you have a set (one ball) – disjoint (two balls). Then the balls turn colors, grow hairs, or whatever, in my head as they put more conditions on. Finally they state the theorem, which is some dumb thing about the ball which isn’t true for my hairy green ball thing, so I say, ‘False!’"”
Bret Victor’s description of what “really good programmers” are doing is also related:
“[showing the code for binary search] In order to write code like this, you have to imagine an array in your head, and you essentially have to ‘play computer’. You have to simulate in your head what each line of code would do on a computer. And to a large extent those who we consider to be skilled software engineers are just those people who are really good at playing computer.”
I think when we imagine an LLM as a human who’s read everything ever written in chess but never played an actual game, we’re actually tricking ourselves - because that hypothetical human would be ‘playing chess’ inside their head by imagining the pieces and moving them according to the rules they had read[1]. LLMs are not doing anything like that when they read about chess. So it’s a very restricted (or perhaps more accurately, a very different) kind of ‘reading’ that we don’t have any intuition for. Since the ‘reading’ that we do have an intuition for is smuggling in exactly the kind of “modeling the world” ability we’re looking for, it’s not surprising that this argument would incorrectly lead us to believe we’ve found it in LLMs.
1: In fact the very best computer chess is achieved by AlphaZero which was trained exclusively on “playing chess in its head”, and it beats even the most powerful and optimized search algorithms like Stockfish looking 20 moves ahead.
What LLM's are doing when they imagine playing chess is what we do when we stand up after sitting on the floor, or what we do when we see a few million individual samples of color and light intensity and realize there's an apple and a knife in front of us.
I think what is almost impossible for most people to understand is that AI's do not need to be structured like the human brain and use the crutches we use to solve problems the way we do because evolution did not provide us with a way of instantly understanding complex physics or instantly absorbing the structure of a computer program by seeing it's code in one shot.
Also, there is no reason to believe that playing chess in our head is anything else but us pattern matching a mental process on a higher level, recognizing a simulation there, and feeding that info back into the loop below. Nature provided us with a complex, layered and circular architecture of the brain, but the rest is pretty much training that structure. And we know that different architectures with similar outcome are possible, since there are vast variations across our own species, and other species as well, with essentially the same capabilities.
I am quite skeptical of these arguments along the lines of “imagine a human read everything written on the topic…”.
What humans are doing when they read something is not what neural nets are doing when they read something. Humans are (idealistically) doing something like Feynman’s description of how he reads (or in this case, listens to) a theorem:
“I had a scheme, which I still use today when somebody is explaining something that I’m trying to understand: I keep making up examples. For instance, the mathematicians would come in with a terrific theorem, and they’re all excited. As they’re telling me the conditions of the theorem, I construct something which fits all the conditions. You know, you have a set (one ball) – disjoint (two balls). Then the balls turn colors, grow hairs, or whatever, in my head as they put more conditions on. Finally they state the theorem, which is some dumb thing about the ball which isn’t true for my hairy green ball thing, so I say, ‘False!’"”
Bret Victor’s description of what “really good programmers” are doing is also related:
“[showing the code for binary search] In order to write code like this, you have to imagine an array in your head, and you essentially have to ‘play computer’. You have to simulate in your head what each line of code would do on a computer. And to a large extent those who we consider to be skilled software engineers are just those people who are really good at playing computer.”
I think when we imagine an LLM as a human who’s read everything ever written in chess but never played an actual game, we’re actually tricking ourselves - because that hypothetical human would be ‘playing chess’ inside their head by imagining the pieces and moving them according to the rules they had read[1]. LLMs are not doing anything like that when they read about chess. So it’s a very restricted (or perhaps more accurately, a very different) kind of ‘reading’ that we don’t have any intuition for. Since the ‘reading’ that we do have an intuition for is smuggling in exactly the kind of “modeling the world” ability we’re looking for, it’s not surprising that this argument would incorrectly lead us to believe we’ve found it in LLMs.
1: In fact the very best computer chess is achieved by AlphaZero which was trained exclusively on “playing chess in its head”, and it beats even the most powerful and optimized search algorithms like Stockfish looking 20 moves ahead.