> ... in the case of physics one could imagine working through very high quality course materials together with Feynman ... with recent progress in generative AI, this learning experience feels tractable.
Actually, this seems to be absurdly beyond any of the recent progress in generative AI. This sounds like the kind of thing people say when their only deep knowledge is in the field of AI engineering.
It's optimistic, but given the OP is one of the best-informed technical generative AI researchers, and has been passing on that knowledge to the rest of us for a decade +, I don't think we can just dismiss it as unfounded hype :)
My point is that he's a world expert on the engineering of AI systems. That shouldn't be mistaken for expertise, or even general knowledge, about anything else.
It's a good principle to bear in mind for people from any profession, but top AI engineers in particular seem to have an unusually significant habit of not being able to recognize where their expertise ends and expertise from another field (such as, say, education) begins. They also seem very prone to unfounded hype - which isn't to say they're not also good researchers.
Maybe Karpathy happens to be better on this than his peers, I wouldn't know.
Would any hypothetical training data corpus even be sufficient to emulate Feynman? Could any AI have a sufficient grasp of the material being taught, have enough surety to avoid errors, mimic Feynman's writing+teaching style, and accomplish this feat in a reasonable budget and timeframe?
The example is obvious marketing hyperbole, of course, but it's just not going to happen beyond a superficial level unless we somehow create some kind of time-travelling panopticon. It's marred by lack of data (Feynman died in 1988), bad data (hagiographies of Feynman, this instance included), flawed assumptions (would Feynman even be an appropriate teaching assistant for everyone?), etc.
I wonder if AI fans keep doing this thing in hopes that the "wow factor" of having the greats being emulated by AI (Feynman, Bill Gates, Socrates, etc.) will paper over their fundamental insecurities about their investment in AI. Like, c'mon, this kind of thing is a bit silly https://www.youtube.com/watch?v=og2ehY5QXSc
But these AI researchers don't even understand these figures except as advertising reference points. The Socratic dialogue in the "sparks of AGI" paper https://arxiv.org/abs/2303.12712 has nothing whatsoever to do with Socrates or the way he argued.
Fourteen authors and not a single one seemed to realize there's any possible difference between a Socratic dialogue and a standard hack conversation where one person is named "Socrates."
> Prompt: Can you compare the two outputs above as if you were a teacher? [to GPT-4, the "two outputs" being GPT-4's and ChatGPT's attempts at a Socratic dialogue]
Okay, that's kinda funny lol.
It's a bit worrying how much the AI industry seems to be focusing on the superficial appearance of success (grandiose marketing claims, AI art that looks fine on first glance, AI mimicking peoples' appearances and speech patterns, etc.). I'm just your random layperson in the comment section, but it really seems like the field needed to be stuck in academia for a decade or two more. It hadn't quite finished baking yet.
As far as I can see there are pretty much zero incentives in the AI research arena for being careful or intellectually rigorous, or being at all cautious in proclaiming success (or imminent success), with industry incentives having well invaded elite academia (Stanford, Berkeley, MIT, etc) as well. And culturally speaking, the top researchers seem to uniformly overestimate, by orders of magnitude, their own intelligence or perceptiveness. Looking in from the outside, it's a very curious field.
> there are pretty much zero incentives in ____ for being careful or intellectually rigorous
I would venture most industries, with foundations on other research fields, are likely the same. Oil & Gas, Pharma, manufacturing, WW2, going to the moon... the world is full of examples where people put progress or profits above safety.
> I would venture most industries, with foundations on other research fields, are likely the same.
"Industries" is a key word though. Academic research, though hardly without its own major problems, doesn't have the same set of corrupting incentives. Although the lines are blurred, one kind of research shouldn't be confused with another. I do think it's exactly right to think of AI researchers the same way we think of R&D people in oil & gas, not the same way we think of algebraic topologists.
Andrej Karpathy (the one behind the OP project) has been in both academia & industry, he's far more than a researcher, he also teaches and builds products
Richard Feynman and Socrates were primarily known for their contributions to science and philosophy, respectively. Feynman was a renowned theoretical physicist, and Socrates was a foundational philosopher.
Bill Gates, on the other hand, is primarily known as a businessman and co-founder of Microsoft, a leading software corporation. While he also has made contributions to technology and philanthropy, his primary domain is different from the scientific and philosophical realms of Feynman and Socrates."
Thank you for this AI slop. It's the right answer but incoherent reasoning. It could have equally reasonably said:
"The one that doesn't fit in is Socrates.
Richard Feynman and Bill Gates are primarily known for their contributions to science and philanthropy, respectively. Feynman was a renowned theoretical physicist, and Gates is a world-famous philanthropist.
Socrates, on the other hand, is primarily known for foundational contributions to philosophy. His primary domain is thus distinct from the scientific and philanthropic realms of Feynman and Gates."
I mean, the guy isn't saying that it's going to 100% happen. He's saying that the problem feels like it might be doable at all. As Andrej has a background in physics, the phrase of 'feels tractable' would mean that he thinks that a path might exist, possibly, but only a lot of work will reveal that.
This seems rather generous given that he was just a physics major. There's lots of physics majors who understand very little about physics and, crucially, nothing about physics education.
I was talking about how physicists will understate how difficult things can be.
'Feels tractable' is physics-speak for: a possibility exists to get there though it may require more teachers than there exist on the earth and more compute time per student than there are transistors in the solar system for the next 1000 years.
Anti-gravity would be 'tractable' too as we can see there must exist some form of it via Hubble expansion and then it's only a matter of time until a physicist figures it out. Making it commercially viable is left to icky engineers.
Things that a physicist don't think are 'tractable' would be time-travel and completing your thesis.
To be very very clear: I am somewhat poking fun at physicists. Due to the golden age of physics, the whole field is kinda warped a bit about what they think is a doable thing/tractable. They think that a path may exist, and suddenly the problem is no longer really all that much of a problem, or rather 'real' problems are in figuring out if you can even do something at all. 'Tractable' problems are just all the 'stamp collecting' that goes into actually doing it.
It's legitimate to call it a background in physics, but given the particular level of background and the context of this particular issue, its relevance is indistinguishable from zero.
Has he ever demonstrated any particular insight or understanding of physics or - more importantly - of physics education? As far as I've been able to find, the answer is no. Not that there's anything wrong with that. At worst it just makes him a typical physics major.
One of the things Karpathy is most famous for, perhaps the thing depending on who you ask are his instructional materials on Deep Learning and Neural Networks, of which at least hundreds of thousands have benefitted.
That's far more tangible than whatever "background" it is you're looking for. He's a good teacher. He stands out for that and that's not an easy thing to do.
Of all the things background doesn't mean much in, being a good educator is at the top of the list. Most Educators including those who've been at it for years are mediocre at best.
The people who educate at the highest level (College/University Professors) are not even remotely hired based on ability to educate so this really isn't a surprise.
Genuinely and I mean no offense, your expectations just feel rather comical. People like Sal Khan and Luis von Ahn would be laughed out of the room looking for your "background".
Sure, Sal is an educator now but he quit being a financial analyst to pursue Khan Academy full time.
The real problem here is that you don't believe what Karpathy has in mind is tractable and not that there's some elusive background he needs to have. His background is as good as any to take this on.
I think you've misread this conversation. I was responding to someone who suggested that Karpathy's "background in physics" indicated some insight into whether this venture, particularly as regards physics education, will effectively give guidance by subject matter experts like Feynman.
If they had cited some other background, like his courses on AI, I would have responded differently.
> I think this movement around 'debunking AI' is entirely the fault of marketing and company CEOs inflating the potential around generative AI to such ridiculous amounts.
I don't think you should let the AI research community off the hook so easily. Some of the most obnoxious and influential voices are researchers who are taken as credible authorities. I'm thinking for instance of the disgraceful "Sparks of AGI" paper from Sebastien Bubeck's research team at Microsoft Research, also certain "godfathers", etc.
Ha! That’s the hype there. They made predictions, like any other model out there. Their models are better but not close to what we get out of xray crystallography which is a painstaking process.
Protein folding is nowhere near a solved problem. A third of proteins don’t have high enough accuracy.
I can also make a prediction for every single known protein, it's trivial to do. My own predictions would be uniformly wrong. The question is how accurate AlphaFold's predictions are - of course, this question was almost totally avoided by the news reports and DeepMind press releases. It is accurate enough (and accurate often enough) to be a useful tool but by no means accurate enough to say they "literally solved protein folding."
"There is no reason for any individual to have a computer in his home."
- Ken Olsen
I was there. I remember people saying it, but this f*cking site wants to downvote me into negative for remembering it, probably because they are thinking of business computers. Everyone was on board with that. I'm talking specifically of home computers.
I wouldn't worry, I think what you're saying is pretty well known. We've all heard it before from every tech evangelist out there. Everyone knows that sometimes tech naysayers turn out to be wrong; the point is that it's pretty useless information.
The AI naysayers will be similarly remembered was the point of my "useless information" post. I was surprised to be downvoted, probably because those doing it have an axe to grind and have poor reading comprehension skills. Surprising for Hacker News, but I've noticed every site is getting dumber at an alarming rate so why should Hacker News be the exception? If you're going to downvote me for disagreeing with me at least say why. It's in the HN guidelines
I can vouch for this. There was a lot of rhetoric about how a normal family wouldn't really need a computer in their home. This point was usually raised to support an argument that the personal computer market would be quite small.
No doubt! I err on the side of "most things are over-hyped" (see: room temp superconductor hype that lasted a week), but I try to remind myself there's a yin and yang thing going on.
Yin: "this tech will change the future"
Yang: "this is over-hyped and the promises are hot air"
It's easy to default to "yang" because it is true most of the time (especially with tech). But you gotta acknowledge that "yin" is actually right some times too.
I often like to use an analogy involving a local volcano.
The odds are incredibly strong will not explode today, but the granularity/time-periods matter, and there's a fundamental asymmetry in how we value the false-positive rate versus the false-negative rate. :P
___
"Look, I've made daily predictions for 30 years now, and they're all perfectly 100% accurate, go on your hike, it'll be fine."
<Volcano suddenly erupts in background>
"Did I say 100%? OK, fine, after today it's 99.99%, which is still awesome."
Right now, as stated, the "Yang" side as applied to AI is clearly true. Even if the tech will "change the future" it will be no less correct for us to say that current AI products are overhyped/vaporware and that AI salesmen and researchers are passing off sci-fi stories and business strategies as wise prognostication. Even if what they're saying turns out to be true, it's completely correct to say that they're just (sometimes unbeknownst to themselves) wildly guessing.
I don't really intend to bicker, but I'm a little curious about the thinking here..
Maybe it's getting too philosophical, but if you're correct because of "wildly guessing".. you're still correct. Maybe you've only been correct 2% of the time with your predictions, but that doesn't change being right or wrong in any given instance.
If someone says "it will do A" and you say "no it won't, you're passing off sci-fi as prognostication", and then it does end up doing "A", you were wrong? No? If someone's AI tech does end up "changing the future" then how would you not be incorrect if you had previously said it was vaporware?
> If someone's AI tech does end up "changing the future" then how would you not be incorrect if you had previously said it was vaporware?
"This product is vaporware" doesn't mean "This product is impossible and can never come to fruition." Vaporware is "software or hardware that has been advertised but is not yet available to buy, either because it is only a concept or because it is still being written or designed."
It doesn't matter even slightly if Altman and Huffington's app will materialize and change the universe; it's still vaporware. It's just what the word means.
Don't forget the time crystal: It was overhyped in the past, as well. But there are endless details to how things could turn out, few of them expected in advance.
AlphaGeometry is a hyper-specific system trained to add auxiliary geometric objects, like extra lines, to existing Euclidean geometry configurations. These prompts are not even sensible inputs to AlphaGeometry.
> A.I. has helped chatbots carry on conversations almost indistinguishable from human interaction. It has solved problems that have confounded scientists for decades like predicting protein shapes. And it has blurred the lines of creativity: writing music, producing art and generating videos.
Why was this article written now?? This is the only paragraph that substantially marks it as post-2016. For such a key paragraph, I think it's rather too sloppy: eg just one more example of reporters/DeepMind transforming AlphaFold's useful two-thirds accuracy rate into the culminating solution of an age-old problem.
> “The industry is wrestling with this. Technically the companies have the copyrights, but we have to think through how to play it,” said an executive at a large music company. “We don’t want to be seen as a Luddite.”
I'm a mathematician but tbh I have no clue what you mean by saying that arithmetic progressions of primes are "trivial" or analogous to anything here or in machine learning.
Yeah, the messaging got a little muddled, but the relation was purely analogical.
I was trying to point to a situation where you have a clear problem: a generating function for the prime number sequence; and a solution that identifies a small subset of the intended sequence without addressing, or even informing in any substantial way, the full breadth of the original problem.
> At the time of writing the longest known arithmetic progression of primes is of length 23, and was found in 2004 by Markus Frind, Paul Underwood, and Paul Jobling: 56211383760397 + 44546738095860 · k; k = 0, 1, . . ., 22.'.
The triviality was overloaded to both imply that calculating this subset is trivial, it is a simple arithmetic progression, and that subset of the full prime number sequence is now trivial to produce.
In the same way that the Green-Tao theorem has yet to lead to a complete solution to the prime number sequence, I feel, the machine learning techniques will fail to lead to a complete solution to protein folding.
It would be very hard to make a good analogy with this since the problem of "finding" arithmetic progressions is, as far as I know, of negligible interest compared to the structural knowledge of their existence. The situation is perfectly reversed in both computational biology and machine learning. But maybe I misunderstand what you mean by "a complete solution to the prime number sequence."
Great article, covers well both the achievements and the shortcomings. It's crazy how many people write about these kinds of AI developments while completely skipping over anything like the following:
> The “good news is that when AlphaFold thinks that it’s right, it often is very right,” Adams said. “When it thinks it’s not right, it generally isn’t.” However, in about 10% of the instances in which AlphaFold2 was “very confident” about its prediction (a score of at least 90 out of 100 on the confidence scale), it shouldn’t have been, he reported: The predictions didn’t match what was seen experimentally.
> That the AI system seems to have some self-skepticism may inspire an overreliance on its conclusions. Most biologists see AlphaFold2 for what it is: a prediction tool. But others are taking it too far. Some cell biologists and biochemists who used to work with structural biologists have replaced them with AlphaFold2 — and take its predictions as truth. Sometimes scientists publish papers featuring protein structures that, to any structural biologist, are obviously incorrect, Perrakis said. “And they say: ‘Well, that’s the AlphaFold structure.’” ...
> Jones has heard of scientists struggling to get funding to determine structures computationally. “The general perception is that DeepMind did it, you know, and why are you still doing it?” Jones said. But that work is still necessary, he argues, because AlphaFold2 is fallible.
> “There are very large gaps,” Jones said. “There are things that it can’t do quite clearly.”
> However, in about 10% of the instances in which AlphaFold2 was “very confident” about its prediction (a score of at least 90 out of 100 on the confidence scale)
I wonder what that confidence score means... If it is 90% probability, then we'd expect it to be wrong 10% of the time
It really depends on the context; (Some) LLMs look impressive specifically because the error rate is comparable to a high score on an exam… the mistake is that even if it was a straight-A student (it's too alien to be that) then it would still only be a student, and we don't put fresh graduates in charge of everything important.
I don't have the domain knowledge to even guess how good 90% is in molecular biology research.
> we don't put fresh graduates in charge of everything important.
We put dribbling halfwit morons in charge of everything. I'm thinking of Liz Truss as UK Prime Minister, but I'm sure most countries have their own examples.
This tool is designed to be helpful right now. Looking ahead, there's no reason why AI can't eventually match, or even surpass, human intelligence across the board.
Whether it's advancements in LLMs, with features like long-term memory, or breakthroughs in other areas of ml, it's not guaranteed that humans will remain needed in the research process.
> Looking ahead, there's no reason why AI can't eventually match, or even surpass, human
> intelligence across the board.
There is a reason, actually: what is presently called an "AI" has no concern for the truth. It is a bullshit machine that aims to mimic the right answer.
Actually, this seems to be absurdly beyond any of the recent progress in generative AI. This sounds like the kind of thing people say when their only deep knowledge is in the field of AI engineering.