Is this something that changed much between versions of GPT-4? Do you think Yan meant that GPT-4 failed with some special private test set in the same style (e.g: to avoid contaminated tests)? I couldn't identify a "monte test with modified elements", do you mean the "Monty Hall problem"? (Not sure that's a basic reasoning test, but it's the only thing I could think of). I didn't see that in the transcript either though: https://lexfridman.com/yann-lecun-3-transcript Or do you mean that by adding some Monte Carlo magic to the test's completion sampling that it can pass just fine?

For the Lex interview I only managed to catch the first parts about image processing, which did seem perhaps a bit dated. I mean to watch the podcast soon though.

Thanks for your thoughts, sorry I didn't get it yet!