> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field.
No. You've fallen for exactly the con the article is describing.
ChatGPT knows nothing. It is very good at generating text that seems plausible if you know nothing about the subject matter, but is not reliable as an actual source of information.
If you happened to get information from it that turned out to be ok, you were lucky. Don't expect your luck to last.
I was surprised to find ChatGPT does a better job of translating than Google translate, at least for one difficult task I needed to get done.
I have been a Google Translate user for more than a decade. I use it a dozen or more times a day for multiple languages. It is a very useful tool.
I recently had to translate a passage that included this phrase: 六六大顺
Google Translate gave me: Liuliu Dashun (basically no translation, just a transliteration or romanization)
Bing Translator: Six Six Dashun (a little bit better because 六 does mean six in some contexts.
Baidu: Liu Liu (basically no translation)
Naver: Six or Six Dashun (partial translation, plus added an incorrect particle)
Only ChatGPT gave the correct answer: "Everything going smoothly"
There is a cultural context that is essential to the translation, which apparently only ChatGPT has. It turns out that the number 6 is a lucky number in Chinese. It has the meaning of being “auspicious”. In Chinese culture, there is an idiom called “六六大顺 (liù liù dà shùn)”, which means “Everything comes in a smooth way” or more naturally "Everything going smoothly".
BTW, after seeing the ChatGTP results, I found this contextual info using Google search, in a website http://www.ichineselearning.com
It is ironic that Google search "knows" (helps you find) the cultural context but Google Translate apparently does not leverage this info.
Seems far more likely to me that ChatGPT knows the meaning of specific idioms rather than knowing the cultural context of that number and using it in the translation.
This passage has some nuance and wordplay, due to the stylistic device of starting each of the key phrases with a character that can represent a number: 四 (four), 五 (five), 六 (six), 十 (ten) and 万 (ten thousand). These characters also have meaning as words within each phrase. For example, 十 can convey perfection, in addition to the numerical quantity.
So here is Google's lumbering and broken translation attempt:
"Wishing you good luck in the new year of 2023,
safe all year,
Five Blessings,
Liuliu Dashun,
perfect,
May all go well with you,
good luck
Auspicious Year of the Rabbit"
Baidu does a better job (including handling the problematic 六六大顺 phrase:
"I wish the new year 2023 a good year,
Four seasons are safe,
Five blessings are at the door,
Everything goes smoothly,
perfect,
Everything goes well,
Good luck,
Auspicious Year of the Rabbit"
Neither of these is as good as the result from ChatGPT:
"I wish you a smooth wind and rain in the new year of 2023,
peaceful seasons,
five blessings at your doorstep,
everything going smoothly,
perfection in all things,
everything going according to your wishes,
continuous good luck,
and a lucky Year of the Rabbit"
ChatGPT is not perfect. I think the phrase 四季平安 is better translated as "four seasons of peace" (this one came from the translator from Korean company Naver).
Yes, one would think. However, if it were a matter of specific idioms, the other translation programs would capture this (Google, Bing, Baidu or Naver). But, for some reason, none of the others did. I think perhaps it is not that common an idiom. Or perhaps ChatGPT was just lucky.
Correct. Eight is by far the most common lucky number in Chinese, but six is also lucky. But I think it has to be repeated multiple times. Apparently the number 9 is also lucky.
GPT Chat: 10-4 is a phrase commonly used in radio communication that means "message received and understood." It is typically used to acknowledge a message or request.
I think that's unnecessarily pessimistic view. I have recently started using ChatGPT for mid-hard questions (when I can't get an answer immediately using Google), mostly "how do I" questions. It lies of dreams up fake answers about half of the time, but this still means that 50% of time I get my solution without digging through obscure manuals.
> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field.
I like to think about it like asking questions to a well read but dishonest student during an oral exam. They know some things, and try really hard to answer every question, even resort to making things up in hope of getting a passing grade.
So it's a coinflip whether it's giving you correct information or something completely made up? And now you're not digging through the obscure manuals to actually verify? Seems objectively harmful.
It seems to me the utility of the 'good' 50% depends entirely upon your ability to recognize it.
How do you know when to dig through those obscure manuals?
It seems to me that it'd be more useful when:
a) I don't care whether the text says things which are not true
Write me a poem about ...
b) I care, and will have to verify it's results, but that cost is worth it
Generate some test data ...
I think we're at a very dangerous intersection between an apparent decline in American's ability to detect nonsense and an automated way to create convincing nonsense.
How many hours you spent trying ChatGPT out? I spent at least high tens, maybe even hundreds. You're absolutely wrong. Yes, it hallucinates, yes its wrong about obscure topics - but calling having success with it luck is absolutely wrong. It's very consistently good. Especially about things like programming, physics, math - and now I'm using it as my teaching assistant for my pilot training, it's perfect (and I can very simply verify the answers are good with my FAA Pilot Handbook, don't worry).
> I can very simply verify the answers are good with my FAA Pilot Handbook
Thank you for agreeing with my point.
If you need to check the answers with your FAA Pilot Handbook, wouldn't it be simpler just to read the FAA Pilot Handbook? That handbook, unlike the text generated by ChatGPT, was written by a process that was aware of the semantic relationship between the text being written and the things in the world that the text was referring to. That is what makes the FAA Pilot Handbook a reliable source of information about flying. ChatGPT is not any such process. That's why it is not a reliable source of information--as you agree, since you need to check what it says about flying with the FAA Pilot Handbook.
> If you need to check the answers with your FAA Pilot Handbook, wouldn't it be simpler just to read the FAA Pilot Handbook?
No, absolutely not. It's much easier to verify a couple specific pieces of information that you're unsure of then go hunting through a large corpus of information trying to pick out the bits that are specific to what you want.
I've used ChatGPT across lots of different areas and I find it incredibly useful. I'm not blindly trusting what it spits out, but it's pretty simple to verify what it's saying. While I definitely do have concerns about the impacts of ChatGPT on a societal level, and what will happen when so much computer-generated content can flood the Internet, but, at a personal level, the complaint that ChatGPT "bullshits with confidence" is not really that much of a problem for what I use it for.
Edit: To give a specific, real-world example, there was a post recently about using ChatGPT to replace a SQL Analyst. Now, ChatGPT definitely will and did churn out wrong answers, but it was incredibly useful to use as a starting point for some complex queries. When it fails, it tended to fail in pretty obvious ways, and to the complaint that it can fail in more subtle ways that look correct, I've certainly dealt with tons of human-generated queries that had the same issues. Are those all useless?
I work with all available material in many different ways (Anki cards, videos, different explanations of the same thing, etc), and ChatGPT is another way to learn and help me generate learning material. For example I have it ask me questions like a tutor would. Or I ask various questions when I'm unsure about the wider context - e.g. it provides much more about the underlying physics than the Pilot Handbook itself. If I don't understand, I can ask for clarification, or an explanation like I am 5.
Reading the Pilot Handbook is a big part of learning but being limited to it would be hard. I'm very happy about having Chatgpt available.
Perhaps you and the poster are approaching your evaluations from different points of view. I've found that if I set out to break ChatGPT I can very easily do it. If my goal is to look for mistakes or find a failure case it's almost trivial to do so.
At the same time if I'm looking for success I normally find it.
Essentially if you work cooperatively with the tool then you'll find it useful, if you are antagonistic towards it you can also have success in breaking it.
I asked it to write some simple code do do a task. It confidently told me to use a library and use some functions in it.
Couldn’t get it to work. Couldn’t get any examples out of google of it being sued that way. Eventually looked through the code of the library and found that while some functions worked int he way chatgpt was trying, the functions it had selected didn’t work, didn’t support those arguments, and never had.
> Especially about things like programming, physics, math
I routinely find ChatGPT giving me completely made-up APIs and incorrect explanations when it come to programming. And I haven't found it much better with math. Sorry, I don't buy it. Maybe it's good at training pilots, and if so great, but it's wrong enough for me that it's hard to trust in general.
> If you happened to get information from it that turned out to be ok, you were lucky.
This is just saying that sometimes ChatGPT is right and sometimes it's wrong, with an implicit stance on the relative frequencies.
In my own experience it would be more accurate to say you're unlucky if it gets it wrong—but then much of this hinges on having a good sense of what it's going to be good at / useful for.
(The other thing I see people miss all the time when assessing its correctness: it is somewhat "on demand"—it may include bits that are vague and only correct-ish, but if you point out an error or request refinement you can generally get it to clear/tighten things up. It's a bit like using a Google Maps-like zooming UI, but in addition to being variably abstract it's always approximating: the potential for error is ever present.)
> This is just saying that sometimes ChatGPT is right and sometimes it's wrong
No, it's saying that ChatGPT is not reliable. It's not reliable because getting things right or wrong is not part of its design at all. All it does is generate text that matches a pattern. There is nothing anywhere in it that filters or evaluates text based on any semantic connection with anything else.
That's another way of saying sometimes it is right and sometimes it is wrong...
> All it does is generate text that matches a pattern
This is about as meaningful as saying, "all things are is collections of atoms".
We are barely beginning to explore the implications of these pattern-extending structures: their most interesting features emerge from their structure, cannot be simply derived from the fact that they do pattern continuation in the same way you're not gonna derive the concept of friendship from analyzing chemical bond structure.
There's lots of real knowledge encoded in ChatGPT and it can recombine and regurgitate it in enough ways that I'd be okay saying that it knows stuff. The real problem is that it doesn't know its limits and will make shit up instead of saying "I don't know". How to reliably detect and enforce these limits probably requires a completely different approach to AI.
I think ultimately any truly intelligent entity needs to have agency - to be able to interact with the world and test truth (prediction correctness) for itself.
Without this you're really just an expert system - a bunch of canned data/rules.
The strength of ChatGPT over a GOFAI expert system like Cyc is how broad it's knowledge base is, but the weaknesses are that is doesn't know whether this "knowledge" (being self-deduced from the training set) is true or not, nor does it have any reliable way to combine facts/sources, since those rules are also self-derived and unverified. I'm sure some of this could be mitigated by more human curation (which seems to be the path OpenAI is going down), but then it seems it is just an expert system (maybe this is all people are looking for anyway).
Of all the languages one could ask ChatGPT to generate, bash would be nearly the bottom of the list. The number of subtle footguns in bash are through the roof.
So frustrating to see people complain that it provides wrong information. With this logic it can never know anything. It is not made for querying. It's made to complete the text you provide it, with some minor modifications to help it answer questions. It does know the most common patterns of characters in the internet which implicitly contain knowledge.
That's correct: ChatGPT does not know anything. That's not what it's built to do.
> It does know the most common patterns of characters in the internet which implicitly contain knowledge.
No, it doesn't. It knows patterns of text in its training data, but it knows nothing about the semantics of the text--its relationship to other things in the world, which is what is involved with any text that contains knowledge. That is why ChatGPT is not reliable as a source of information.
I share your skepticism of LLM’s output but I don’t think it’s fair to say they know nothing about semantics. I think it’s still an open question as to what degree LLM’s encode a coherent world model. Also you can just ask chatgpt about objects and their relationships and it gets the answer right way more often than you’d expect by chance, so it has some understanding of the world. Not good enough for me to trust it though
Not that I see much evidence of what I'm about to assert in ourselves, but you should be able to correct ChatGPT's knowledge if it knows things and isn't just a fancy parrot.
No. You've fallen for exactly the con the article is describing.
ChatGPT knows nothing. It is very good at generating text that seems plausible if you know nothing about the subject matter, but is not reliable as an actual source of information.
If you happened to get information from it that turned out to be ok, you were lucky. Don't expect your luck to last.