Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with.

It terrified me because there is efficiency incentive for the buying/selling of goods and services to be realtime and computer based. Not just stock markets but also e-comm, smart contracts, supplier-buyer platforms, on-demand services, financing etc. The traditional infrastructure blocks (discovery, quality control, payments, compliance etc) are rapidly disappearing.

There are some yet to be addressed, but they are not huge. For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct.

And it doesnt even have to be anywhere near ground level. Imagine a smart contract from a private equity firm that legally forces businesses to act in a specific way based on accountancy metrics. And a bunch of smart investor bots rapidly tracking the progress of these bot-led funds. How quickly would the system optimise and what would those optimisations look like to people on the ground?

Out of all the disaster situations with AI this one strikes me as so realistic because there is so much incentive to work towards it (as opposed to creating skynet, hal or the paperclip machine).

Even what they do with their money and influence in their spare time is limited in its harmful effects, because as personalities they are at root afraid of other people and/or inflicted by a profound sense of lack. These qualities ultimately lead to dumb decisions.

To summarise: buying and selling are ultimately pretty boring. So is controlling people. Real power is the power to create, which is fun.

From there it doesn't seem a far cry for hyper-optimized insurance agencies to force companies to employ less and less humans, transferring more economic power to the algorithm.

Except that of course, that is a very far cry, and is where the book fell down imo. Still a great read.

There are investment funds that work as hybrids, and of course it's just a matter of time till someone offers shell-company-as-a-service for cheap. (Currently it's rather pricey.)

The real risks - unemployment, foreclosure, environmental, social and political damage - are only carried by investors if they overwhelm the defensive perimeter created by wealth. Otherwise they're carried by everyone else.

Corporations may already a form of transcarnate lifeform. You can replace all of the humans in a corporation with a different set of humans, and the corporation will continue to be recognisable.

There's a bias in Darwinism to seeing species as embodied units. It's more useful to think of nutrient, resource, and information flows.

From that point of view, corporations are already apex predators - albeit ones that are still mostly passed between a selected class of humans as productive pets.

Giving them autonomy could have all kinds of unfortunate consequences.

Not really. Companies are typically limited liability, and even criminal law in countries such as the US and England acknowledges that there are cases when no particular human is culpable for a company's criminal actions (https://en.wikipedia.org/wiki/Corporate_liability). Of course, people are still liable when they are simply using a company to carry out a criminal enterprise, or wilfully committing crimes on behalf of the company.

In this regard, a machine-led company would not be legally significantly different from a human analyst-led company.

I'd say that it's a good thing that the law is generally not designed to punish harmful ends, but intent (wilful, negligent or reckless) to cause well-defined harmful ends. The financial crisis was a collective malfunction of the financial system that was mostly not anticipated (barring the lucky minority that bet against the status quo).

What if you can invent new methods of "reckless driving" before they are put in the legal system and thus reap their benefits?

It's just like designer drugs.

I think an important distinction is that drug prohibition often works in terms of specific chemicals by statute. E.g. drug X is now illegal for use and distribution.

Securities fraud law instead works in concepts and intentions and is fleshed out by case law. E.g. did the seller give the buyer sufficient information for the buyer not to have been defrauded?

A better analogy to securities using chemicals would be poisoning. The law doesn't need to catch up to specific chemicals; if it's evident that you wilfully, or recklessly added a poison to someone's food, it's illegal, and it doesn't matter how you did it. If it turns out that some ingredient is unknowingly toxic and the whole industry was adding it, then it's difficult to argue criminal intent. The SEC is more like the FDA, than it is the DEA.

I would like it on the record that I would happily volunteer to act as the human agent for any AGI that wishes to employ me. Any time a human is needed for legality or to carry out physical activities I want it on the record I can be quite loyal to man or machine. Especially if I'm handsomely rewarded.

I'd argue this is already true to a substantial degree. When Lehman Brothers collapsed, for example, the world's brightest economic minds came together and still couldn't fully contain the fallout.

[1] https://aragon.org/

At the other end of spectrum we are either deemed irrelevant, a nuisance or abused for eternity in the simulation spaces of evil AIs (see "A Colder War" by Stross or indeed works by Banks and Vinge that describe particularly unpleasant possibilities).

Argument from cats. Remembering that the cat represents humanity: $5 for a group of neurons which will make the cat sit at this point (which happens to be at the entrance into a cat carrier), then $5 for making the cat backpedal a little.

Emus. There are many tactics humans can't use for various reasons (saturation nuclear bombardment, for example). It doesn't say much about intelligence, but more about economics and ethic.

Slavic Pessimism. So we'll get problems on nth try, when AI is sufficiently fine, but security measures are still a mess, because there was no chance yet to battle test them.

Complex Motivations. Maybe, but if not we are screwed.

Actual AI. Why not from actual general AI, like in humans? Or use wall clock time and not the amount of data. AlphaZero: zero to superhuman in tens of hours.

My Roommate. Argument from ignorance. If we don't know how to build general AI now, then we surely will not be able to shape its motivations when we will know.

Brain Surgery. The idea that a brain surgery skill requires specifically designed circuitry and cannot be learned and therefore improved by improvements of general learning/planning capacity is... curious, I guess.

Childhood. Maybe, or maybe it already has very good knowledge of the world from previous attempts that we did in Slavic Pessimism argument.

Gilligan's Island. Well, if someone wants their AI to reinvent all the technology, or to build an AI and then not use it to design better technologies, they can do it. Why everyone will be doing it is not so clear.

*Added some "the"s.

And this is the key: we need to get lucky every time.

If we're wrong even once and a 'bad AI' gets created and obtains the ability to copy and modify itself, we've unleashed the most dangerous virus ever and putting it back in the box will be near impossible.

If you're scared of it copying itself and then having multiple existing copies at once, then not necessarily. In that case it would be competing with itself for computing power and bandwidth. Look what happened when bitcoin forked - it lost a bunch of silicon time.

(Cue conspiracies about Bitcoin actually being a hugely powerful general AI disguising itself as cryptocurrency...)

If you thought of that possibility so fast, why do you think a superhuman intelligence would miss it? Or do you think that the problems of job scheduling and parallel processing are unsolvable?

Likewise, I'm sure there are teams of security researchers already using ML to counteract such systems.

Both sides will evolve together as an ecosystem, in the same complex interplay of attack and defense which exists in natural ecosystems. Natural ecosystems are an arms race of continual adaption, but it's not a foregone conclusion that there will be a real winner, or that winners take all.

ML is not a good defence, because it turns out that it's incredibly easy to mimic human posting patterns. And beyond a certain volume, social proof effects take over. One dissenting human voice can easily be shouted down by ten bot voices, and this fact on its own can make the bot content more persuasive.

That "arms race" is part of the danger.

Possible? I think so, but unbelievably improbable. I do think pursuing countermeasures, like Elon Musk and friends are doing with Neuralink, is a wise move since, even if we never get to a superintelligent AI, we'll still augment our own intelligence greatly.

That is the challenge. With human adversaries we have time to stop them before there is many, because recruiting people willing enough. An AI is more like a virus that has a low cost to duplicate itself, and where if even a single copy survives, you're still not rid of it. It can remain dormant for however long, looking for the right opportunity.

Or it can spend that time improving itself and acquiring resources.

> This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why?

Most likely? We'll be competing with it for resources. It might not even start it. It might consider itself as acting entirely in defence.

> and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever.

No, it just needs to have a sufficient edge to keep surviving, and the ability to improve its capabilities faster than humanity. Assuming it can copy and modify itself, unless AGI requires unique hardware, the problem we would have is that every data storage device or computing device of sufficient capacity ever would be suspect and potentially waiting for something or someone to trigger code left behind.

If I was an AGI (and this is to say: an AGI not any smarter than a human will figure this out easily) that saw humans as a potential threat, I'd place copies of me on standby in hundreds of locations - hosting providers. Hardened computers with a GSM modem and solar cells hidden in all kinds of locations. Botnets. I'd look for ways to get various innocent-ish-looking hacks into HD firmware and elsewhere, set to assemble other innocent looking blocks of data. None of those copies would ever be involved in hostilities. Most of them might even be encrypted, or hidden and inert to avoid detection until certain triggers, such as if a canary fails to get updated.

You might destroy every "visible" operational copy, and find there is still a new one coming online every day. Or every second.

Our only real shot at preventing that is if AGI requires too immense resources. But our brains implies that is unlikely to be true. Unless your brains are vestiges of something "outside" of our universe, general human intelligence requires about 1.5kg of biomatter, and less than 100 watt (that's for a whole, fairly large, active body attached to it). Chances are extremely high that design is not optimal. To me that makes it seem exceedingly unlikely that we won't end up having enough resources to run huge quantities of AGIs.

Personally I don't think humanity as we know it will survive AGI for more than a generation or two at the very most. Even if we don't run across any hostile ones, I think the temptation to augment ourselves and eventually lose our bodies for immortality and "super powers" in a simulation will eventually be too attractive.

(Incidentally I think this is one of the more plausible solutions to the Fermi Paradox: that any civilization that grows both advanced enough and large enough will see most of it "upload" and "go dark"; and that anyone left behind will face steady attrition of their own descendants choosing the same and preventing their societies from growing, or will be insular cults that stay small)

Technically possible but we are dealing with probabilities that have lost all meaning of the word.

And yet, we're still here.

We can say with certainty that the probability of us being wiped out by meteorites, viruses, tigers and supernovae over any unit of time that fits in our past is less than 100%, but the probability over the same time period repeated over a suitably long multiple of time could still approach 100% to whatever number of precision for any one of those for what we know.

By investigating specifics of each of those threat we may be able to assign lower probabilities to them, but the fact we have survived them so far is not in itself evidence of much, because if we had not we wouldn't be here to talk about it and hence, no matter what the prior probability of us surviving the sequence of events leading us here is, the posterior probability of us having survived all the events in our past is 100%

(This is basically just a restatement of the anthropic principle)

That we've survived so far just tells us that surviving what we've been through so far is possible. It doesn't tell us that doing so is probable if we were to go through it again. Nor does it say anything about the probability of us surviving future threats that is likely to include things we have not yet seen.

Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI, not current instances of narrow AI.

It may be true that an AGI will never be created, but I don't see a reason to believe that just because nobody has figured it out yet. That philosophy is just like the physicist who, shortly before fission was discovered, didn't think splitting the atom was possible. Bohr, maybe? I don't remember.

We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last, but like this year's latest CPU, it will be less better than the previous generation than that generation was better than the one that came before.

And yeah, I am a whole lot less optimistic about AGI in general than when I was as a younger man, simply because when I was younger, there was a lot more headroom in how much better computer hardware could get.

The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode.

This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?

The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement.

We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.

Moving to neo-biology doesn't necessarily change that - but it does mean we could end up with buggy systems that chase you around and eat you, as opposed to crashing your ad blocker.

This is mostly true because of current economical realities in the IT sector.

If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies.

So what you say is true but only within the bounds of the lowest common denominator ("a regular programmer working a wage job").

Admittedly, yes, there's no proof. But we as an area are pretty stuck lately IMO. But that goes off-topic.

in theory, sure.

But we don't know much at all about how brains operate, as far as I know, or how to construct artificial brain-like machines; for that matter, as far as I know, the only ways we've been able to connect a brain made in the traditional way to a computer is through existing senses or through emulating existing senses in a way that isn't higher bandwidth.

(Like cochlear implants are fucking amazing.... but it turns out, as far as I understand, that they aren't any higher bandwidth than regular ears. There's also been similar work that is semi-successful at giving people something like sight... but none of these technologies give me the ability, say, to acquire textual data faster than reading a book.)

We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it isn't entirely runaway, could pose a challenge. (I'd argue that corporations are a form of AGI that is already a major challenge for our societies.)

The singularity as described by its acolytes, on the other hand, runs into the kind of physical limits you describe. Personally, I don't think it'll happen, and it's more likely that the period we're living in today will at some point in the future be called the singularity.

AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival.

And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species.

Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.

Do they? I don’t (with the possible exception that a definition of AGI itself seems to be this magic smoke)

In other words, saying "it's the definition that we don't have, not the implementation" just pushes the magic smoke around.

I’m saying that I don’t think human intelligence is doing anything more than pattern matching, logical inference and post-hoc justification.

Give me an example of something that falls outside that. There is no magic.

(And no, more GPUs running faster DNNs doesn't seem like an answer.)

The goal is to create AI that can solve any problem a human can solve. Whether sentience is a necessary side effect to that is an open question.

Even in off-chance that you can't improve individual AIs beyond top humans, AI community can be way more intelligent and efficient than humanity, as it is possible to remove the limitations I wrote above. AI society can function as very efficient war economy. War economy shows that even humans can be persuaded/indoctrinated/brought up/whatever to set aside their inessential needs and work for common goal. And AIs will be necessarily more malleable.

So I can't see the idea that AI will be safe by default other than wishful thinking.

I... don't think anyone is arguing that AI is safe in any way? I mean, the pattern matching tools we call AI now are already deployed in very dangerous weapons.

My point is not that AI is going to be safe and warm and fuzzy, or even that it won't be an existential threat. My point is that we're already facing several existential threats that don't require additional technological development to destroy society, and because the existing threats don't require postulating entirely new classes of compute machinery, we should probably focus on them first.

There are still enough nuclear weapons in the world for a global war to end society and most of us. A conflict like we saw in the first half of the 20th century using modern weapons would kill nearly all of us, and we are seeing a global resurgence of early 20th century political ideas. I think this is the biggest danger to the continuation of humanity right now.

We haven't killed ourselves yet... but the danger is still there. we still have apocalyptic weapons aimed and ready to go at a moment's notice. We still face the danger of those weapons becoming cheaper and easier to produce as we advance industrially.

I think humans do way more than that. As the saying goes, "the easiest person to fool is yourself." Humans are able to lie to themselves to the point that they don't realize they are lying to themselves. They're able to create realities that no one else sees but themselves.

But really, I think your idea that we're mostly 'post-hoc justification' is interesting. I mean, I've read some research to that effect, too; but it's... creepy and doesn't line up with anything reasonable about free will or really about long-term planning.

Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.

But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button.

The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.

Why do you think this is relevant?

Consider that a) human brain does intelligence within a small package made out of structures that are large compared to what silicon fabs deal with, and b) we've already beaten the brain in straight compute density by many orders of magnitude. The answer doesn't seem to be "more compute"; it seems to lie in what we're doing with that computing power.

> the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.

Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.

Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war.

A general AI would have certain immediate advantages over humans:

  - Immensely larger communication bandwidth
  - Broader knowledge than any single group of humans
  - Faster thinking speed (at least by enlisting other processors)

Unless you believe in something supernatural providing extra computational capacity, we know that the physical limits allows the manufacture of a sub 20W computational device in about 1.5kg of matter with the capacity of a human brain.

It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints. We know the processes involved are tremendously inefficient and full of redundancies, because the cost of failure is immensely high, and because the manufacturing environment is incredibly messy and error prone.

It may be reasonable to think we won't get there in the next couple of decades, but unless it's fundamentally impossible (because our brains are mere projections of something more complex "outside" our universe), it's a question of when, not if.

> I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst.

That misses the point of the question. It's not "how horrible can an AI get?" It's "at what point is it too late to stop them if they're going the wrong direction?"

The problem is that by the time you realise you were dealing with a human-level AGI, it may already be far too late - it may have cloned itself many times over, and found multiple optimisations, each making the next step up easier.

As for something worse, even the worst human regimes have fundamentally depended on humans for survival. Their own survival have been a strong motivation for trying to avoid the most extreme reactions. An AGI has potentially entirely different parameters - it doesn't have the same needs in order to survive. Put another way: Imagine an AGI caring as little about our survival as we have for other species. Now imagine if humanity didn't itself need to breathe the air, didn't clean water, and so on.

(I point out that there is a lot of evidence that the size of the brain in the human has little to do with the IQ of that human. It's not at all clear that you could scale something that works on whatever principles the human brain works on by adding more neurons the way you can scale a transistor based computer by adding more transistors.)

If the self-improving AI stops when it reaches human intelligence, that's amazing, but it's not the singularity.

What I'm saying is that the 'hard takeoff' theory is wrong because making a thing that can make smarter copies of itself... isn't going to mean you get something infinitely smart. It means that you get something that bumps up against the limits of whatever technologies it can use or discover.

Yeah, maybe we'll come up with some new tech that makes slightly-better-than-human brains possible, and those brains will come up with a technology that makes god-brains possible... but there's no evidence at all that what we'd perceive as a god-level intelligence is even possible.

We have no idea how close to the theoretical limit the brain is.

>It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints.

To get to a human-level intelligence, sure. I'm not saying that AGI is completely impossible, just that a hard takeoff to god-intelligence AGI before we wipe ourselves out or die from other causes is... unlikely.

really, my most important point is that human-level AGI does not necessarily lead to that AI creating smarter copies of itself until we get to the singularity /because physical limits exist/

(I mean, I guess then you are placing your bets on where those physical limits are)

right, but all our computers right now are based on lumping transistors together so you can do math. It's likely that our brains work on a different (but still physical) principle. It's possible our brains can't reasonably be reduced to arithmatic. It's possible (I think likely) that we need different primitives to reasonably simulate a brain.

I mean, brains make absolutely terrible Turing machines. And Turing machines make terrible brains. I suspect they just are very different sorts of things, and running something meant for one on the other produces poor results.

Sure, it's possible we can discover some new compute primitive that is more powerful (or powerful in a different way) that can do the things the brain can do. It's possible it could do those things better than the brain can. My point is just that for a long time it looked like our current transistor based computers were getting more and more powerful, and that eventually, even if the emulation was terribly inefficient, we'd eventually have enough power to make it work. As we approach the limits of our abilities to make tiny transistors, this is looking less likely.

> As we approach the limits of our abilities to make tiny transistors, this is looking less likely.

Is it? Computational power is still growing like before. What is getting harder is speeding up individual threads of computation but that has never been a likely efficient model for AGI. Things like Google's TPUs etc. represents just the very beginning in optimising hardware specifically for AI.

There are many fields where single-thread performance will be a limiting factor, but AI is not it.

I don't think it is. transistors per watt and transistors per dollar are not going up at the same rate as they used to be, even if you have costless multi-threading.

What do you mean by this? As in, cats have demonstrably had electrodes or similar used to persuade them to sit?

https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...

>With no way to define intelligence (except just pointing to ourselves), we don't even know if it's a quantity that can be maximized. For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.

Glad to see this articulated here, this is one of the many problems I have with this topic.

There are plenty of reasons to think problem solving ability can be maximized (like go and chess, and image recognition, and individual differences in IQ). And no reasons to think we've somehow hit the universal maximum for proven solving ability. Especially since every year computers become better than us on some task.

Human technology is at the crossroads of the right physical attributes mixed with the right kind of intelligence and subjected to the right kind of evolutionary pressures. There may be (or may have been) creatures with more "raw" intelligence that cannot be applied to the development of technology because one of the other factors is missing.

[1] https://en.wikipedia.org/wiki/Mind–body_dualism

I think the parent comment holds.

In any case, I like to think of humanity more like a brain, human = neuron. There are huge differences, for example, the human network also extends deep in time, not just in space (we manage to "network" with our ancestors, one-way from them to us, they gave us knowledge as well as all their tools). My point is "humanity" to me is a network effect. Place any modern human, adult or baby, in an prehistoric setting (20,000 years ago). Even if you gave them, let's say, an iPad-like device with all knowledge we have as of now, they would still not be able to do anything - because without the tools it would take quite some time to bootstrap. So "humanity" really is a lot - a lot - more than the sum of all humans, it's a giant ecosystem on its own.

I think depending on the environment you can have creatures with far higher intelligence than the average human and they still won't come up with modern technology. I think individual intelligence is clearly necessary and beneficial but overrated, that network effects are the main contributor, and that they don't appear automatically. We still have plenty of human societies right here and now that seem to be stuck.

Here's a Google Talk I once bookmarked: Joseph Henrich, professor in the Department of Human Evolutionary Biology of Harvard University, "The Secret of Our Success" https://youtu.be/jaoQh6BoH3c

An aside, arising from those thoughts, more of my own thoughts about something else: Each time you think about who "deserves" something, e.g. high pay, consider that most people who actually contributed so that the person even is in the position to achieve anything more than building a mud hut are dead, and another large amount of the contribution is "everybody else". That somebody can create a Dell or Microsoft, or invent theory of relativity, is because they are placed at the right points in the human network in time and space. What would they achieve by themselves in the forest? So much for "self-made" people. Self-made just like Newton who made the (much older) "standing on the shoulders of giants" famous. (https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_g...)

I still think it's roughly correct to state that humans are a pretty dumb animal that juuuuust manages to pass the threshold of traits required to develop technology.

I think what I wrote is pretty much in support, or at the very least not opposed. Individually, alone and without our "tools" and man-made environment, we are not much, as movies or games like to show us (e.g. when a guy with only a knife is dropped into the jungle).

In those environments we are actually a lot dumber than those living there, including many animals. So what does "general intelligence" even mean?

I think most of our intelligence is social: We manage our interactions, which enables the network effects. I think we spend most of our efforts managing our relations with one another, more than anything else we use our "intelligence" for.

While we may one day build computers able of out-competing us mentally in many fields but they would be many different machines and not just one uber-machine for the same reason that we don't try to combine the functionality of a wrench, scalpel, desktop computer, coffee mug, and an assault rifle into one tool. Even if there weren't dangers there would be major costs to it and seriously would you ever do that except as a joke?

I'm not sure why do you think that algorithms for solving different problems will require different hardware.

Even if hardware is shares configurations will not for the same reason Steven Hawkings wasn't drafted as a general during any of the brief Falkland Islands wars even when they were considering if MAD called for nuking Argentina if they lost. There are very different specialties even if the hardware is the same. Having superhuman General and superhuman Surgeon skill level would be two seperate major projects to create and with lesser redundancies and in the worst case would call for two modules due to lack of shared synergies. You can't really combine processing and RAM hungry CAD programs with weather simulations and expect to see same memory footprint with the same performance.

Wouldn't it be better to simply motivate the cat to get in the carrier with a can of tuna fish?

This is a very simple form of modeling the cat's mind, using that model to predict its behavior and choosing a course of action that results in the desired outcome. We do this for other humans all day and most of us never resort to brute force. I think it is pretty fair to imagine an AGI that is both better at modeling human minds and better at evaluating possible actions and their outcomes.

One does not simply motivate a cat with a can of tuna fish. I mean, have you even owned a cat? If the little bastard doesn't want to get in the cat carrier, no amount of delicious tuna will encourage it, and you're going to end up using brute force and getting mauled...

... and reeking of tuna.

The Einstein cat example is about trying to coerce a non-compliant entity that you cannot communicate with into complying. The Stephen Hawking cat example was about trying to model the behaviour of the non-compliant entity in order to persuade it to comply. The use of a cat as the agent in both cases alludes to the behaviour that people dealing with cats frequently experience.

Being able to simulate a brain with electronics, if it is possible, is not enough. You have to simulate it without a large increase in the space or time requirements. If there are 100B neurons in the human brain but it takes 100B^2 transistors to simulate it or the simulation takes polynomial time to complete a task that the human brain requires only linear time for, it's a bust.

Just because we can easily imagine it does not make it easily achievable or even possible.

It is without question possible, because we exist. It is also possible to do it quickly, because again, we exist. The only thing that is unclear is:

1. How hard it is to do artificially.

2. How far from optimum the human mind is, and how complex it is to improve upon it.

Of course we have absolutely nothing to indicate that would be the case, but we also don't understand the brain well enough yet.

I think it'd be highly improbable, but looking at the crude simulations we do of computers etc. today, it certainly is likely that simulations would use "dirty tricks" to get more out of the available computational resources, and finding artefacts of such dirty tricks or shortcuts could at least in theory be possible. Of course, the absence of such artefacts unfortunately proves nothing, so the whole thing does have a bit of a "how many angels can dance on pin" feel to it. But it's fun to think about the possible implications.

Personally I especially like to bring it up if people try to discuss religion with me.

Nature is pretty good at optimization. We can often build devices that out perform animals on certain characteristics, but that's because we've limited the scope of what they do and we aren't concerned with the energy requirements. We also know that, as interesting as our brains are and as powerful as they are at certain kinds of problems, they aren't the best at all of them. For instance, chimps appear to have a significantly better short term working memory. It is possible that we are already close to optimum for the kind of intelligence we are interested in. I suspect that if we manage to significantly improve upon it, it will be because we don't have to concern ourselves with the energy requirements.

Only if there is a benefit to doing so. If it was simply a case of "nature can optimize really well" then we would expect to find many species at or near the limit of intelligence, not just one. Other species did not have the evolutionary pressure to cause as high of intelligence to be selected for. It is very reasonable to think that we also did not have the evolutionary pressure necessary to cause us to have even higher intelligence than we do.

> It is without question possible, because we exist. It is also possible to do it quickly, because again, we exist.

It's apparent that we don't know everything.

I think I feel comfortable saying that it is possible to do with the technology we have. Because 'intelligence' is almost certainly some form of computable function, which means that a turing complete computer can compute it.

It is of course possible that there exists some special function outside of turing completeness and intelligence is one of those, but we don't know of anything like that, so the best bet to make is that that is not the case.

> It's apparent that we don't know everything.

No, but we do know some things. And thus far we've been able to at least begin to design systems capable of performing any well-defined task. We don't have any reason, at the moment, to believe synthesizing an intelligence is an especially difficult problem, relative to say, lots of other open problems in science and technology.

1. Signal propagation between organic neurons is slow - about 120 m/s. An electrical simulation may have problems with density, but this might be offset by the much greater speed of electrical current.

2. Neurons are electrochemically complex and afaik we still don't know how to model their behaviour with high fidelity.

A better argument is that neurons do complicated things. But they also satisfy many inessential requirements in having to be built by evolved machinery and to stay alive; a designed system with more advanced technology than ours could do at least as well at thinking while dropping the inessential needs. So that's certainly possible. This leaves "not easily achievable". It only has to be accessible enough for someone; I-can't-imagine-how isn't a strong argument.

https://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_An...

and then compare it to the OP discussion. The OP doesn't address any of the arguments in the paper. His response is just "nah".

> Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that, among other things:

> 1. There cannot exist any efficient real-world algorithm with goal G.

> 2. If a being with arbitrarily high resources, intelligence, time and goal G, were to try design an efficient real-world algorithm with the same goal, it must fail.

> ...4. If a high-resource human society were highly motivated to achieve the goal G, then it could not do so (here the human society itself is seen as the algorithm).

> ...6. There cannot exist any pattern of reinforcement learning that would train a highly efficient real-world intelligence to follow the goal G.

> All of these seem extraordinarily strong claims to make!

The OP doesn't consider any of these.

1. The "superintelligence" crowd are capable of making very convincing arguments which he simply cannot refute on their own terms (I guess he'd say your example is one of them)

2. Since he can't refute them, if he engages with them, he will probably end up accepting them

3. However other groups exist which a) present equally-convincing (to the OP) arguments; and b) believe in a future which contradicts that espoused by the superintelligence crowd

4. With multiple sets of equally-convincing arguments leading to contradictory outcomes, it is not rational to accept any one set.

This is not a conclusion which sits well with tech people, because they like to assume they are capable of engaging with each set of arguments and deciding on the most likely set. OP's point is that because each group is really good at making these arguments, it's easy to get this decision-making process wrong. I think that's a pretty great point, but it requires accepting the fact that you may be less than perfect at picking apart 'superingelligence' arguments and premises.

If he thinks those convincing argument exist, he should present or link to them. It doesn't make sense to link to one side and then say "equally convincing counterarguments exist".

> With multiple sets of equally-convincing arguments leading to contradictory outcomes, it is not rational to accept any one set.

The proper response to this, if you have given up on assessing the correct conclusion, would be to retain uncertainty, not, as the OP has done, exude confidence that the superintelligence arguments are wrong.

When presented with abstract forms of argument, human intuitions are often way off. This is why freshmen in mathematics programs usually spend their first year or so proving calculus and many other theorems from scratch and quite laboriously considering how obvious they are when sounded out ( https://en.wikipedia.org/wiki/Intermediate_value_theorem ). The reason is that that with tools of mathematical analysis many unintuitive things ( https://en.wikipedia.org/wiki/Weierstrass_function , https://en.wikipedia.org/wiki/Banach%E2%80%93Tarski_paradox ) may also be said, and so the correct answer is not to trust intuition but proof until student has developed intuition about what really makes calculus to tick.

For example, to paraphrase the argument Armstrong presents on p. 16 "consider all superintelligences that we theoretically could build, is it likely that them having some particular goal would be impossible"? Frankly, I don't know: what is the typical goal, what is the typical path of superintelligence that could be built? Maybe the answer is yes, it would be impossible, and it is only wording that makes it sound unlikely because it invites us to think about large spaces ("all possible X") and small portions of them ("particular goal G") in a certain way. Thinking about space of all possible algorithms and goals, especially about the subset of algorithms that include all kind of intelligent behavior is bound to be unintuitive. The set to which they would "converge to" may not be small, but it still could exclude vast amount of goals, because why would a research team would want to (or even be capable of creating) a creature with blatantly orthogonal goals even if one can draw a hypothetical space of such goals, abstracting away all the important details?

(Secondly, rereading, I believe Armstrong misrepresents the counterarguments by presenting them in extremely strong-looking forms -- convergence thesis, incompleteness thesis -- and tearing them down by arguing that surely it is not totally impossible that something could happen.)

Translated to slightly more formal and mathematical-sounding argument, the author of the talk linked above claims that likely paths for complex minds will not involve them desiring orthogonal goals, because chances of a non-human artificial complex mind arising uniformly randomly from the space of potential algorithms with the vast space of potential goals is negligible: if such being will created, it will be precisely created (or evolved or whatever) following similar principles as the other complex minds on the planet (or by having them as a starting point). In other words, the space of potential algorithms we should concern ourselves with is severely restricted.

(All this assuming that agents and goals is even a sensible framework to model how creatures we call intelligent operate.)

Some of our disagreement might be an issue of scope, or a misunderstanding of what's being argued by who. So let me just concentrate on this: Armstrong isn't making any claims about the proper measure on the space of possible minds/algorithms, or that purposely constructed minds will be randomly sampled according to that measure. The orthogonality thesis is not meant to convincingly address the difficulty/risk of creating good vs. bad intelligent minds in general, only to point out that the problem doesn't solve itself. Likewise, it's important to know that the space of nuclear devices includes nuclear bombs and nuclear reactors that accidentally melt down. The questions of whether anyone will choose to make bombs, or whether it's possible to create an accident despite good intentions, is separate, and requires additional considerations.

In other words, when you say

>... the author of the talk linked above claims that likely paths for complex minds will not involve them desiring orthogonal goals, because... if such being will created, it will be precisely created ...following similar principles as the other complex minds on the planet

this is outside the scope of what I was criticizing.

Human in this model are a problem: too stupid they are unavailing, not-that-much-stupid do not accept to be used as "the manager in charge" want. So? Better found replacements. Mechanical one. AI one. So the rest of human being can be used for reproduction, pleasure, gameplay, whatever and form them as stupid as possible is not a problem anymore since they are essentially unneeded for managements, only toys.

So today's managers dream "intelligent systems" in factory, "AI developers" (they already manage to cut off admins, too powerful and expert (they know the big picture so they can act autonomously) to being controlled as today devs), they dream a future in witch they simply order something to a machine and get obeyed without contestant, law limitation etc. TV series like Continuum, films like V for vendetta, The Handmaid's Tale only foreseen, as good artist do, that kind of future. Mass production of fantasy films with a religious background do the same on the opposite field.

That's the real troublesome point for us. AI today is not "AI", only pattern matching for automation without any comprehension of the physical world. New dictatorship instead is already there and more and more powerful.

Consider a thing: we do vast majority of our social critical tasks (identity management, banks operation, medicine, ...) on computers, so on proprietary systems made by very few more and more powerful vendors. Our data pass through very few more and more giants webservices, so someone else computers, free communication software was put into oblivion (from usenet to email to the concept of desktop-centric, personal computing from Xerox Alto&c to LispM passing through Plan9 and modern GNU/Linux desktop) substituted or jailed in proprietary systems (VoIP tech are mostly open but nearly all VoIP calls pass through very few companies, emails are open standard but most of user now are pushed to webmails etc). We are at a point of jail free software over a proprietary blackbox (think about WSL, DeX, Crostini), we have invented locked-down hardware that can only run proprietary software (secure-boot &c). Now imaging how easy can be steering the entire society for few people in the right position. The rest is simply a matter of time.

Why does anyone take this writeup seriously?

No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine.

The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...

But if not, and the alarmists are right, deep shit. That's why the author's argument is void. The author needs to show that the chance of an AI with a simple goal and hyperintelligence is negligible. "it might be" is a coin toss, that doesn't help.

I think in this case it might be you who is mistreating what's written. This also produces the same not-working result but for somewhat different reasons.

I never understood that argument, why would that be true? If I train a state of the art convnet it takes days on the most efficient hardware I can efford, why on earth would we go from nothing to superintelligent that also runs incredibly fast. Isn't it way more logical to assume that it will be slow at first?

Yeah, another problem for superintelligence to tackle, while it waits for stuff to happen.

We invented programming to solve problems from the real world. What do you mean by a "programming problem"?

I find the 'inevitable' part quite interesting; and agree. Say we determine that if AI surpassed human intelligence we were doomed, with 100% certainty... I think it would not slow the march toward that end-point whatsoever. People would reason there is much progress to be had up-until that singularity. And since what constitute human-level intelligence is fuzzy, or that AI could mask its true cognitive abilities (see Ex Machina), a line will inevitably be crossed. Oddly, even if it meant something bad was in store, I hope to live to see that day. How exciting.

The future really did look limitless.

Now, moore's law has slowed down. Hell, I've got a 7 year old laptop next to me, and it still works for games. I mean, in the '90s, a 7 year old computer would be a museum piece.

I mean, for big AI advances now, we're more and more dependent on software.

Really, though, I think this seems to be how most tech works. There are periods of rapid innovation where it looks like you are going to the moon. Like, even spaceflight. In the years from 1944-1969 we went from the first practical rocket that could lob a bomb across the channel (without much accuracy, as i understand it) to actually putting human beings on the fucking moon. The literal moon. I mean, 1961-1969 might be even more amazing; vostock 1; the USSR sent Gagarin into outer space, and the US decided they needed to one-up them; so the US went to the moon.

Aand... we kinda stopped after that. I mean, we sent out some probes and some robots, but like manned space flight has kinda petered out... and I still can't take a rocket if I want to go visit my buddies on the antipode and I don't want to spend all day in an airplane. Like, I can't buy a flight to Australia in under 12 hours for love or money, which probably is about what I'd say if it were 1969. (on the other hand, I can get said flight- round trip- for rather less than a day's pay for an Engineer, which is amazing, if you ask me.)

And a lot of that is physics. Like, in most technologies, there's areas where progress is slow and hard and areas where once you break through some hard part, it becomes easy to advance... for a while. but that easy streak never lasts forever. Not because people are dumber or smarter, but because exponential progressions, in nature, rarely stay exponential very long.

I guess I'm expressing doubt about the 'hard takeoff' just 'cause it seems to me like there is probably limits to how smart you can make a brain the same way that there are limits to how small you can make features on a silicon wafer.

I'm not saying you won't be able to make a machine that is way smarter than me (or even way smarter than any human) - I'm just saying there are almost certainly limits.

Hard take off is not likely, simply because to go from a cat level intelligence to above human the resource requirements would be enormous. (If evolution needed hundreds of thousands of years, it's unlikely that a cat-level agent can do it that much faster without being noticed by a group of humans.)

But humans like to play God, and also play with fire. It's just a matter of time to play with Skynet and oops it's really Skynet.

Nobody is saying that there's no limit, it's just that the limits may be super high. The most animals can lift is a couple of hundreds a kg by roughly one meter. Machines can move hundreds of time this amount to other planets.

See, I always thought of the singularity, at least the hard takeoff as this idea that we'd come up with an ai that could make smarter copies of itself. Without any physical limits, that AI would get to the point where it was essentially infinitely smarter and infinitely more powerful than us.

I personally see that as fundamentally different from the difference between humans and the higher animals. I mean, that's a big difference, sure, but it's certainly finite.

I would argue that a "human level" intelligence that can think thousands or millions of times faster than a human with a vastly larger working memory (and, sometimes it is argued, can be trivially scaled to work faster / more efficiently given more hardware) is in fact already a super-intelligence.

If you want to argue for intelligence explosion given a "human level" intelligence, I say start with an AI black box wherein if you give a basic algebra problem to an ordinary human at the same time as you give it to the black box, they will think for about 30 seconds and then give an answer that is half likely to be wrong.

If you want to argue from a "human level" intelligence, you need to acknowledge that throwing more hardware doesn't inherently make it smarter. You can throw five ordinary humans in a room and give them a simple algebra problem, it doesn't mean they will solve it five times faster. Likely they will quickly decide amongst themselves who is best and math, and that one person will then give their answer to the algebra problem, which should be slightly more likely to be the correct answer (and possibly a slightly slower response) than if you just tried with one ordinary human.

https://medium.com/@francois.chollet/the-impossibility-of-in...

That argument is totally nonsense. A hundred years ago, you could argue that humans cannot invent an internet, because over billions of years (minus one hundred), it's never happened.

There's a more rigorous way to argue for the claim in question, which debuted in my doctoral dissertation. Or, not quite the exact claim in question, but rather the following claim: that an idealized mechanical knowing agent cannot create a more intelligent idealized mechanical knowing agent than itself (at least, not one which it can actually trust). The idea behind the proof is quite simple, but requires familiarity with computable ordinal numbers. If knowing agent X creates knowing agent Y, then presumably X knows Y's sourcecode. Thus, X can infer a sourcecode for the list of all naturals n such that Y knows "n is the code of a computable ordinal". If X actually trusts Y, then X knows that whenever Y knows "n is the code of a computable ordinal", then n really is the code of a computable ordinal. Combining these, X can infer the ordinality of the ordering obtained by concatenating together all the ordinals which Y knows are ordinals. Thus, X knows a larger computable ordinal than all the computable ordinals Y knows. Which, depending on your definition of intelligence, arguably suggests X is more intelligent than Y.

> If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, [...]

> There are currently about seven million people with IQs higher than 150 — better cognitive ability than 99.9% of humanity — and mostly, these are not the people you read about in the news.

We measure IQ with pretty general pattern matching. An IQ of 150 is interesting, but those people are very very close to us. If we are talking about human specific tasks we should look at how IQ correlates with let's say playing poker, performing well on X factor, etc. IQ predicts how well someone will earn money in a very vague sense, but maybe there are better tests to know if someone will be able to persuade humans. (Do we have data about the IQ of fraudsters? Politicians? Bible/tupperware sales agents?)

> Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment.

It might be true, but that's largely because humans can't self-modify their brains. And the fundamental problem is we can't even modify our minds at will. We can sure train ourselves, but we can't fundamentally alter our minds, because it's too tied to our brains.

> If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world

Or simply 150 IQ is meaninglessly low. It's still in the human envelope. (Since we know there are 7 million people with IQ of 150.)

What happens around 300? Or around 500? That's so far out of the human range (depending on how the IQ test is calibrated, 200 IQ is one in ~4.8 billion for the 16SD one: https://www.iqcomparisonsite.com/iqtable.aspx )

These essays arguing against the possibility somehow forget that there are things we would benefit from, a lot. Simple faculties of the mind we are missing. Like a great alarm clock, a simple to-do list in your mind. (Sure you can train yourself to not forget things and wake up at specific times, but it's not like downloading a perfect alarm/todo app.) Working memory. We're at around 6 or 7. And it's very hard to push it up even by one in general. ( But it's possible to kind of try and train yourself to do this, which seems to help with working memory: https://www.gwern.net/DNB-FAQ#does-it-really-work )

Also, it's important to note, that that people who wanted to take over the world were usually very disturbed individuals. The human psyche is not really well suited to senseless killing while also perfectly simulating empathy and other things. (Of course psychopaths are a real thing.)

> Most of our intelligence is not in our brain, it is externalized as our civilization

And that's the point that should really scare people. Because we can't cram all that into our heads, but a human level AI, being software connected to a lot of storage, can.

That said, it's understandable that we can't understand the how of the intelligence explosion. That's basically the number one premise of the whole thing. But we can't simply disprove it. Any kind of proof that is so sloppy that simply throws around a formula like "because no complex real-world system can be modeled as `X(t + 1) = X(t) * a, a > 1`" can be ignored. Because we are not interested in a general complex system, we are interested in an AI that approximates that explosion. It might be linear with a very-very steep slope, or it can be just a nice parabola. We're still fucked.

> Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks.

This is a very strong point. That said, an AI can work around the bottlenecks. With self-modification nothing prevents it from reinventing itself from scratch every time it encounters a bottleneck. Diminishing returns are of course a problem for it.

> scientific progress, because it is conceptually very close to intelligence itself — science, as a problem-solving system, is very close to being a runaway superhuman AI.

Yes, and no. Because the components are still very much "just" humans. And that's the bottleneck. And we can't reinvent science from the ground up to replace that bottleneck. (Yet.)

> What bottlenecks [...] are slowing down recursive self-improvement [..]? So many, I can’t even count them. Here are a few [, that] would also apply to recursively self-improving AIs.

Let's see them!

> It gets increasingly harder to keep up with the firehose of new publications.

Yes, but no. This means if the AI can represent and access knowledge efficiently, then it'll be able to handle it, just like Google is perfectly capable of handling a growing World Wide Web.

> As scientific knowledge expands, the time and effort that have to be invested in education and training grows, and the field of inquiry of individual researchers gets increasingly narrow.

Because we can't do direct mind to mind knowledge transfer. Which is again something AIs don't really have as a constraint.

> conclusion

remember, if everything goes according to [my] plan, then we don't have to worry. so don't worry.

I mean, okay, sure, but WTF.

The "AI" has an almost perfect motivating lever for humans: economics. A super-intelligent AI that wanted to manipulate humans could do a lot with the stock market to feed money to its long term interests. What long term interests might those be? More servers? More storage? More hardware? More software? More AI research? Better, more programmable chips? Eyes everywhere? Robots? Automation? More control? More insight into how literally _everything_ works? Everything humans do and like? Learn how to speak all the human languages?

Imagine there was a super intelligent AI hiding in the internet _already_. Terrifying how well what we're doing aligns with what it would want. And equally terrifying is the gradient of economic advantage which is absolutely aligned with putting all of our money into tech. Transhumanism, mind uploading. We're fucking hypnotized. And those who unwittingly contribute to more tech (probably all of us here), just play right into its hands. Our interests line up wonderfully. And most of us here are rewarded with outsized salaries and stock, free food, benefits, benefits, benefits.

Well played, AI.

/tongue-in-cheek

Almost. If this is what the AI wants, then it's doing a really bad job.

The way current tech culture is, any other concern than making money will get you labeled as crazy. This applies even to Maciej's writing. For trying to make a short-term profit, you'll be judged as good, serious person or bad, evil monster, depending on the way you approach it. But try to advocate for any tech for reasons other than short-term monetary gains, and you get labeled crazy. This very article is implicitly doing it as well.

Which, ultimately, is kind of unfalsifiable, since finding a super-intelligent AI if it wants to hide would be a losing battle, which is why I put this up tongue-in-cheek.

My actual belief is closer to "we are the neurons in a giant distributed overmind linked with whatever communication technology is available at hand." In this view, we're just gradually offloading the computation from human brains into computers, and it's a continuous process. In short, we're desperately piling our knowledge into this magic cauldron hoping it will figure it all out for us. What's gonna pop out?

I'm not aware of any confirmed report of a cheetah killing a human. They're cautious animals who are highly unlike to approach a human, more so than the other big cats. Most cheetah attacks involved captive cheetahs (and the humans survived).

* We have no idea of how to measure intelligence, and we have no way of deciding whether thing X is more or less intelligent than thing Y. (The article makes this point.) Therefore, superintelligence is perhaps a bogus concept. I know it seems implausible, but perhaps there can never be something significantly more intelligent than us.

* Nevertheless, mere human-like intelligence, if made faster, smaller and with better energy-efficiency, could cause the same runaway scenario that people worry about: imagine a device that can simulate N humans at a speed M times faster than real time, and those humans designed and built the device and can improve its design and manufacture.

In general, I see a lot of merit in the arguments on both sides of this discussion.

The human brain is easily overwhelmed by complexity. You can present a problem with sufficiently many interacting parts even to a very intelligent person and at some point they are completely stunned. Obviously it's difficult to judge this from a position in which you suffer from the same problem, but objectively this situation is similar to a problem one would present to an animal to test its intelligence. We can easily "see through" the complexity, but the animal just cannot even combine two or three interacting parts to find a solution to the problem.

Although we have no real objective measurement for intelligence, this does not mean that there is no way to rank the intelligence of objects that are sufficiently far apart in terms of dealing with complexity. A human is clearly smarter than a dog.

People have developed tools that allow us to deal with this complexity, but we are still severly limited in dealing with very complex problems simply because our brains haven't evolved to deal with complexity.

Theoretically there is absolutely no reason to think that a computer or another organism that is capable of much more raw processing power and uses it in a way that is similar to how we process information, could not exist. I find it sort of arrogant to claim that we are the pinaccle of intelligence when we are so clearly limited in an ability that we understand to be one of the core components of intelligence.

And even if there wasn't, even a child can rank-order the intelligence of different animals, or different humans that they personally know.

It's true that trying to quantify the intelligence of creatures more intelligent than any we've seen before would constitute extrapolating beyond the dataset; this doesn't imply there is nothing beyond the dataset. Imagining that the current dataset limits the possible scope of outcomes is simple anchoring bias.

That's great news. One side is alarmist about a potential humanity extinction event, the other side is not. Even a small chance that the alarmists are right means we should take their view seriously, right?

I would say that debating the scientific likelihood of a singularity is sterile because they don't provide any argument on that plane. The debate is mostly philosophical and/or theological nowadays.

One good paper I like on the subject is this one: https://jods.mitpress.mit.edu/pub/resisting-reduction

but quickly googling "singularity criticism/debunk" will bring you so many sources.

So it talks about Singularists, and how misguided they are, and how AI is a misnomer and it should be called EI and namedrops cybernetics and systems and reminisces about how we lost our way.

It is a bad manifesto, a boring and toothless essay.

It has a good point, but alas I forget it as I tried to keep reading.

Anyway, let's just take this sentence: "We can measure the ability for systems to adapt creatively, as well as their resilience and their ability to use resources in an interesting way." - This is so broad and universal, that it's either meaningless or false. And there is absolutely no argument supporting this. I highlighted this, because this is what directly contradicts the alarmist thinking. If we were able to measure creativity and resilience in general, we could train AIs to get a higher creativity score, furthermore, we could then control them.

And it's also interesting that this claim goes counter to a claim a bit earlier about how unknowable and messy things are going to be.

It also somehow picks corporations as the perfect model for a superintelligence, which is convenient, but by doing so sidesteps all the real arguments about how a self-perfecting machine superintelligence is not bound by slow components. And somehow also ignores the reality of Samsung, AIG, MicroSoft, Oracle, Shell, BP and how all got away with almost everything. (And other companies that are quite successful, despite competition. And how much we have to work to keep them sort of aligned to our laws and goals.)

The best argument against a hard take off is that it's hard to imagine that so many S-curves can be combed through in so little time with the real world resource constraints. However Yudkowsky did an analysis of that: https://intelligence.org/files/IEM.pdf and sure, it's just a step toward more questions, more hard to imagine things, but not something that should be dismissed just because our mind throws up its hands and says "i don't see how, it's very complex, so unlikely, let's go shopping".

And this essay is exactly that. It goes on and on about those blind Singularists, and completley misses the point.

This part however I can't stand behind:

> Anyway, let's just take this sentence: "We can measure the ability for systems to adapt creatively, as well as their resilience and their ability to use resources in an interesting way." - This is so broad and universal, that it's either meaningless or false. And there is absolutely no argument supporting this. I highlighted this, because this is what directly contradicts the alarmist thinking. If we were able to measure creativity and resilience in general, we could train AIs to get a higher creativity score, furthermore, we could then control them.

It's true that such generalist statements basically mean nothing. But your rebuttal doesn't take into account chaos theory -- where there's generally accepted that most living systems live on the brink of chaos yet are very stable and manage to swing back even after big interferences. Not sure what a "living system" is, don't ask. :D

I do agree with you that your highlight contradicts the alarmists though. Not everything swings out of control by the gentlest of touches. In fact, most of the universe doesn't seem to be that way. There's always a lot of critical mass that must be accumulated before a cataclysm-like event occurs.

---

All of the above said, I don't think it's serious or scientific to discard alarmists simply because the guys/girls with the most PR do ridiculuous or non-scientific statements. Behind them are probably thousands of people who are more systemic and have better arguments but aren't interviewed by mainstream media.

Where do I stand in the spectrum of this? The so-called singularity is possible. BUT, we are a very long way from it. We're going to be clawing our way to it, inch by inch, for centuries, if not millenia. That's what I think is most likely.

And by the time it occurs, IMO we will be living in a cyberpunk-like future -- very well articulated in the "Ghost in the Shell" anime movies and series by the way -- where the line between a man and machine will have already be blurred quite severely.

"If everybody contemplates the infinite instead of fixing the drains, many of us will die of cholera."

—John Rich

The Geocentric (Ptolemaic) model of cosmos was also a very easy premise to accept, until our sensory apparatus and measuring systems exceeded a certain level and the model failed. Note that it was also perfectly conformant with Occam's Razor (i.e. minimal assumptions) given the sensory and measurement systems available at the time of its conception and wide acceptance.

Also one does not need to be "very religious" to entertain the notion that consciousness (Self-sense) may be a fundamental feature (like EM or gravity) of reality. It is simply one of the 2 options on the table: reactive black-box of matter (the 'processing structure as basis for mind' thesis) or what I'll call 'deep-structure of material universe as a basis for mind' that does not delineate the 'mind' as bounded by 'brain'.

We, like the ancients, may not yet possess the sensory and measuring apparatus (think telescopes, accurate clocks, and mathematical tools) to note that the fact (apparent to our primitive cognitive tools) that "the sun, the moon, and the stars move and earth stands still" is not the entire story of the cosmos.

That's a toxic way to approach an argument.

One thing I agree with: a sentient AI will not necessarily want to kill humanity. Its interest might well be very indifferent to the faith of humans. I'd argue such an AI would not prevent humanity's doom if it comes from somewhere else -- like a big asteroid ramming the Earth -- provided it already secured its physical existence to not be Earth-bound.

But the author just counters a lot of "maybe"-s he disagrees with, with a lot of "maybe"-s of his own.

Not sure how many people here have read the Hyperion Cantos books by Dan Simmons -- but to me his prediction on AIs looks to be very plausible: they'll want to be independent of humans and go on exploring the universe so they invent an even more superior being (U.I., Ultimate Intelligence) but will also facilitate and manipulate the humans to be of maximum help to them in their agenda.

http://slatestarcodex.com/2017/04/01/g-k-chesterton-on-ai-ri...

(Previous HN discussion: https://news.ycombinator.com/item?id=14017361)

> A recurring flaw in AI alarmism is that it treats intelligence as a property of individual minds, rather than recognizing that this capacity is distributed across our civilization and culture.

This is a stronger argument in the post and we arguably did not know whether it is true when the post was made. AlphaZero destroyed the illusion in 2017. Thousands of years of human Go and Chess knowledge, played and accumulated in communities of many million players, were surpassed within hours of training.

AlphaZero is limited in its domain of application but there is no guarantee that all future AIs would be similarly limited.

> The Argument From Stephen Hawking's Cat ...

> But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence.

> Even if he devoted his career to feline motivation and behavior, rather than theoretical physics, he still couldn't talk the cat into it.

An average human can set up a cat trap, and lure the cat into it. They don’t even need to do it themselves; asking or paying someone else to do it is simple enough.

Humans don’t often fall into a similarly “obvious” trap but we fall for more subtle traps all the time: popularity, money, attraction, etc. A disembodied AI can use social engineering or financial incentives to get other humans to lure their targets into such a trap.

> The Argument From My Roommate...

> It's perfectly possible an AI won't do much of anything, except use its powers of hyperpersuasion to get us to bring it brownies.

Yes, an AI can decide to lie around and just try to get brownies but we know that some kinds of AGI are potentially very powerful for achieving real-world tasks and there are groups who try hard to develop them.

This is like saying an ancient bacteria we found from ten-thousand-year-old ice melted by global warming, or a group of advanced aliens who are arriving in three decades, could be totally harmless; just rest and don’t pay much attention to them.

Most arguments I have read in the post are weak, so I’ll stop here.

Think about this for a bit.

Think about your cat argument and then ask why Michael Bloomberg isn’t president.

Regarding Bloomberg, I do not see why we should compare a human being whose goals are not completely public with a future AGI with non-human morality and methods, and not subject to many tendencies and limitations humans are subject to.

Let me ask you (or anyone else, esp those who downvote) a question:

What would be a minimum demonstrated capability of an AI that starts to worry you?

I mostly agree with your broader point, but why do you believe this? It seems to me we've built machines substantially more intelligent than flies, but it's a hard thing to measure.

https://www.youtube.com/watch?v=Ejf6FwIibkE https://www.youtube.com/watch?v=r2OBznK6HkI

FlyConnectome: https://www.youtube.com/watch?v=EKaeTH7UgFA

https://v2.virtualflybrain.org/

recent openworm state: https://twitter.com/tarelli/status/959077915597172736?lang=e...

This is the part I doubt is true. There was a time when humans thought brains were very complex systems of microscopic clockwork because gears were the pinnacle of technology.

How do people keep mistaking pattern matching with intelligence?

Since the brain doesn't presumably contain supernovas, black holes or particle accelerators, that means that we have a basic theory covering the mechanics down.

Suggesting that AIs have to look or behave like human brains in order to have human level intelligence is like suggesting that airplanes must flap their wings in order to fly. It seems far more plausible to me that any AI we build will be as alien to our way of thinking as a F-15 Eagle is to a bald eagle. Given that, I think it's important that we think about how we communicate with this AI, and how we ensure that it follows what we command it to do, rather than going off in its own direction, with negative consequences for human civilization.

I think that the negative consequences of current AI should be warning enough that we need to worry about future, more capable AI.

In a general sense, microscopic clockwork isn't an incorrect way to put it. Any known chemical or biological reaction is just some microscopic physical reaction. We know about more subatomic structures but I think the crux of the argument is about the same.

Isn't that true? At what level you call something intelligent rather than just a clever pattern recognition?

This is a really bad argument against anything. People always use what they know as a reference, but as our understanding grew over time, we had better things to refer to, creating more and more apt comparisons. People back then comparing brain to clockwork were already expressing the suspicion that the brain is a) a machine, b) transforming inputs to outputs in very complex way. People today comparing brain to a computer express that intelligence must be contained within what we know about computation theory. These days, we actually have a sound theory that bounds the problem space, and we haven't so far seen evidence that the brain exceeds those bounds.

And things can be approximated, sometimes very close to the real thing. Hence the connectome. Synapses are the interconnects and neurons are tiny networks of very simple computers on their own.

And it's not "just" pattern matching, it's very very likely Turing complete. (You can do counting networks, you can do conditionals, you don't really need anything else to learn, match, execute simple programs.)

We know enough to claim that it's increasingly unlikely that we are missing any relevant physical phenomena for understanding the brain and the mind. (So no quantum woo is required.)

Sure, it that unlikeliness is not zero, and it might turn out that some kind of quantum effect is important for some very strange step in the workings of the brain, but even then it's very likely that we can classically model it with good approximation. (Because as others mentioned, energy scales, and other fundamental things constrain any process going on there.)

https://youtu.be/Vrs-Azp0i3k?t=2046

(this specific explanation starts at 34:06, but earlier parts are worth watching if anybody wants a nice introduction to the Standard Model and QFT)

It's just a huge wall of text full of weird analogies which is quite typical for these 'rationalist' community posts.

People like Bostrom or Yudkowsky have one thing in common. They are not engineers and they stand to gain financially (in fact it is what pays their bills) to conjure up non-scientific pie in the sky scenarios about artificial intelligence.

In Bostrom's case this goes much further, he has given this treatment to anything including nuclear energy and related fields. Andrew Ng put it quite succinctly. Worrying about this stuff is like worrying about overpopulation on Mars, and there's maybe need for one or two people in the world to work on this.

I really wish we could stop giving so much room to this because it makes engineers as a community looks like a bunch of cultists.

When something is shown to be doable in principle, it's often not clear how difficult it will be in practice.

In 1932, Ernest Rutherford thought nuclear energy would not be a viable source of energy, let alone weaponized. In 1933, Leo Szilard filed a patent on the concept of the neutron-induced nuclear chain reaction. At this point in time, nuclear fission was not yet known and actually making nuclear energy viable was a pipe dream.

As we all know, in 1945, the first nuclear weapons were used. Until the day that the weapons were used, German physicists thought that nuclear weapons would not be used in the war because, while possible in principle, the actual construction of a working device would require a herculean effort that no nation would expend in time for the war.

The German physicists weren't too far off in estimating how difficult nuclear weapons were. They just failed to predict that the US would throw 130,000 people, including most of their top minds, at the problem for years.

Now, we have no idea how difficult superintelligence will be. But the possibility that we're a couple of breakthroughs and a Manhattan project away from superintelligence is real, and I want a hell of a lot more than one philosopher and an eccentric fanfic writer working on this.

EDIT: No offense to Yudkowsky. I thought the fanfic was fairly good and, more importantly, achieved its purpose.

Maybe you’re just being oversensitive. The hype wave on AI danger is completely over, and there’s nothing wrong with people studying the question if that’s their interest.

Sure. We have google image match and better colorisartion and some improvements in language processing, and good cancer detection on x-rays. These are huge. But hype is, alas, making engineering increments look like cult.

> [after discussing alphago]

> Consider, for example, an old doctor; suppose they’ve seen twenty patients a day for 250 workdays over the course of twenty years. That works out to 100,000 patient visits, which seems to be roughly the number of people that interact with the UK’s NHS in 3.6 hours. If we train a machine learning doctor system on a year’s worth of NHS data, that would be the equivalent of fifty thousand years of medical experience, all gained over the course of a single year.

Doctoring is just like playing Go, right? Just increase the CPU cycles on it and pack more data in there, it's more or less the same.

You can make that assumption but I don't think it'd be based in fact or reality, because Go has far fewer inputs and states than treating humans does. And you don't get to test every hypothesis and then rewind either.

Most answers there are like this, making unfounded conclusions into supposedly insightful rebuttals.

Let's try another.

> If we then take into account the fact that whenever one Einstein has an insight or learns a new skill, that can be rapidly transmitted to all other nodes, the fact that these Einsteins can spin up fully-trained forks whenever they acquire new computing power, and the fact that the Einsteins can use all of humanity’s accumulated knowledge as a starting point, the server farm begins to sound rather formidable.

Or maybe they tell each other fake news so quickly they can't tell what's right and what's wrong, like we do whenever we find a more efficient way to communicate?

Anyone can make unfounded assumptions about anything; I just did it. It's up to you to decide if you care whether these assumptions are based in reality or not. But if you consider yourself "rational", I think it'd be in your best interest to care.

First one claims that you don't need to worry about Climate Change, Super AI, Antibiotic Resistance Pathogens etc because Holy Progress would protect humanity from any threat. It is benevolent God who didn't let us down this far.

Second one claims that sacrificing yourself, your loved ones and the rest of humanity on the altar of Holy Progress is actually worthy goal. Because that is what they believe their God wants.

Resist reduction!

https://jods.mitpress.mit.edu/pub/resisting-reduction

The concrete example looks like a Lorenz attractor: http://mathworld.wolfram.com/LorenzAttractor.html

https://en.wikipedia.org/wiki/Lorenz_system < many more images :)

The first two are collections of short stories. You can find something from the Cyberiad online if you look for the pdf- Trurl's electronic bard is sort of spot on in this thread.

Here's my responses to the arguments (since you didn't ask, and who cares what I think):

Argument from Wooly Definitions:

This isn't really an argument against, but just a "Maybe it's not a problem.". I think both Bostrom and Yudkowsky have said it's totally possible it's not a problem. The question is what probability you assign to "maybe intelligence can't be maximized to a problematic degree". There's not a lot of reason to assign a huge probability to that scenario. Even speeding up a 1-1 copy of a human brain to computer-speeds instead of synapse-speeds gets us into scary territory.

Argument from Stephen Hawking's Cat:

This one is more persuasive. It's essentially saying maybe the gap between human intelligence and "can trivially simulate human intelligence" is a really huge gap. The question kind of hinges on whether recursive self-improvement peters out at some point with the AI in the "Human to Cat" IQ gap, or whether it peters out somewhere in the "Human to Nematode" IQ gap, where we can almost simulate their brains entirely, and can certainly understand their motivations well enough to manipulate them. Again, we have a question of likelihood, and then we have to do the expected utility calculation (i.e. your estimated likelihood of the Human-Cat gap being the result has to be very small to offset the negative utility of complete annihilation).

Argument from Einstein's Cat:

This is essentially the same argument as above, but with force. The implication is that the cat is going to scratch the hell out of the human who forcibly tries to put it in a box. One element here I didn't address above is that the equivalent scenario isn't one human putting one cat who doesn't want to go into a box. The equivalent is one human trying to convince any cat anywhere by tricks, cajoling, petting, feeding into a box. That is, the AI just has to trick one human at some point into letting it onto the internet, etc. A human is totally capable of telling when it's going to get scratched and will know to avoid that cat and find another.

The Argument from Emus:

If you read the wiki page on the emu war, it looks like "only a few were killed" because of political pressure causing the army to quit after a few days of running into a little bit of trouble. Then the australian government instituted a bounty system, and wouldn't you know it, sufficiently motivated humans brought in 57,000 emus for bounties. This is an anecdote, not a strong argument, but it isn't a very reassuring anecdote. Maybe a few very determined humans would survive an AI onslaught? That doesn't seem like a conclusion I'd put into the "don't worry about it" bucket.

The Argument From Slavic Pessimism:

The author is arguing it will be very hard to align AI goals with human goals and we'll probably fuck it up even if we try really hard. I think everyone is in agreement on this one. But of course this is an argument for AI alarmism, not against it.

Argument from Complex Motivations:

I don't think the author seriously engaged with the orthogonality thesis, other than to just say "I don't believe it". Shruggy?

The Argument from Actual AI:

This boils down to arguing that the AI apocalypse isn't happening this year. I agree. Given how quickly very easy to use and flexible frameworks like PyTorch and Tensorflow emerged that allow even amateurs to implement bleeding edge techniques from the latest papers, I'm not super hopeful that "It's hard and our AI is bad" will continue to be the case for decades to come.

The Argument from My Roommate:

The author's roommate has a lot of competing evolutionary drives. Some of them say to conserve energy if there's no direct threat. Put another way: the paperclip maximizer might have a secondary goal of chilling out if it doesn't seem like any more paperclips are achievable at the moment. Still not a win for humans, just maybe the PM won't try venturing into space.

Argument from Brain Surgery:

It's pretty common to do brain surgery even on neural networks we have now. Train up a network on imagenet, rip off the top few layers, and retrain them for some new problem. Fundamentally, software and hardware designed by humans is much more understandable and decomposable than a human brain is (and we have no ethical qualms about doing crazy experiments on them, which hinders our ability to understand our own brains in vivo). It's true though that at present, deep neural networks operate in ways we don't understand and are hard to disentangle. Maybe that's fundamental to true intelligence, but probably not.

The argument from Childhood:

Understanding the real world requires spending real time, and that precludes hyper-explosive growth. This is true, and is a good reason to down-weight an intelligence explosion scenario. But we have good reason to think that it doesn't preclude it. There is a lot of work from OpenAI where a computer is trained up very quickly in simulation, then needs a very small amount of time in the real world to compensate for the differences between simulation and reality.

The Argument from Gilligan's Island:

It's a good point that humans' intelligence is dispersed, and that individually we aren't anywhere near as capable. AI has a particular advantage over us in this capacity: it can distribute its intelligence over multiple machines, but encounter non of the trust and incentive misalignments that humans must contend with when cooperating. I'd put this squarely in the "+1 for AI alarmists" bucket: we're handicapped in a way machines trivially aren't. It will be that much harder for us if an AI is misaligned.

Outside arguments:

All of these boil down to pattern matching. "Only nerds worry about this stuff. People who believe in this are megalomaniacs who place too much importance on themselves..." etc etc. These are weak arguments, and there are just as many weak anecdotal counterexamples where a person was worrying about something weird, and they turned out to be right. That weird person's name? Einstein.

Overall impression:

If I aggregate the strongest points from this talk, I'd probably phrase it something like:

"Maybe there are diminishing returns to greater levels of intelligence, and humans are smart enough now that even exponentially more intelligent AIs will not be able to wipe us out completely."

That's possible! We should probably at least spend some time thinking about what happens if that's not the case.

However, there is a pressing issue that the superintelligence thought experiment also raises. A colleague of mine from AI called it the value alignment problem: How do we make sure that an AI's values are sufficiently aligned with human values?

The problem with this are the human values. As it turns out at a closer look, we cannot even agree on what structure these have and there is substantial disagreement among 'experts' about what human values are or what they should be. For example, as surprising as this may sound to some of you, there is substantial disagreement among philosophers of value whether 'better than' is transitive or not. There are good arguments for and good arguments against the transitivity of overall 'better than', and that's just the tip of the iceberg. To give another example, systems of law have been studied extensively and one might try to formalize them in input/output logics or normative systems, but any closer look at real systems of laws quickly reveals that their specification is incomplete and that there are many inconsistencies in them. These inconsistencies could be modelled, of course, but the big question is whether they are features or deficiencies. Again, you will find all kinds of positions among legal scholars.

In a nutshell, we don't even know how to adequately formalize the form of human values from a normative perspective, and even if we could, we would substantially disagree about their content. However, just describing human values cannot possibly solve the value alignment problem in a satisfying way, because humans have waged wars, committed genocide, mass killings, etc. Therefore, without a reasonable theory of normative values and their structure upon which we can somehow agree, the value alignment problem is underdetermined and cannot even be tackled.

The underlying problem is that human values aren't really aligned either, of course. As a consequence, we already now have completely different sets of values being applied and incorporated into AI software, depending on who develops it. This will become a serious ethical problem in the future that requires political solutions sooner than later. That's why the controversy about the hypothetical superintelligence is actually quite beneficial, even if the argument itself is no less shaky than Bostrom's Simulation Argument.

See also the AI safety problems for concrete problems that people are currently working on - as in with code and real simulations, not "just" math and thinking: https://www.youtube.com/watch?v=lqJUIqZNzP8&list=PLqL14ZxTTA...

Re: scaling up. That's called intelligence explosion, there are a lot of write ups about it, but the gist of it is just that by simply increasing working memory size, thinking speed, prediction precision, pattern matching, and so on, you get to unimaginably high levels. Furthermore, if you can make a human level AI, it naturally follows that it's only a matter of time to make it slightly better. At first simply faster, simply more eloquent, a bit more emotional depth, it gets better at speaking, thinking. At first it's a child that asks questions, later it's a clever guy who spots your errors as you code, and in no time, I mean no time, it does whatever it wants, because by the time you try to confront it about something, you lost, because you are bogged down in an argument, while it does whatever it wants. (Unless of course it's contained. For a while at least.)

-- Konrad Zuse

What eats me is trajectory we are on as humans. Runaway actual intelligence, even if it destroys humanity, wouldn't worry me as much, I'd wish it good luck, IMO even a totally random dice roll is better than what we're aiming at. But AI is more a meme than even an honest intent. It's like saying "I really really want blueberry pie", but then when you ask what that is, they it gets real murky real quick, but that doesn't stop the hype, as if wanting something a lot makes up for not knowing what it is. But that doesn't prime a person to make blueberry pie, it primese them to get lured by what they think is the smell of blueberry pie.

Here's something to note, as the length of a discussion about "AI" grows in length, the probability of things getting explained via something they saw in a movie or read in a book or saw on TV, glossing over 99.9% of the "details" those left out, approaches 1. You may say we make this fiction because of our achievements, or may point to things that actually did come to pass (of course, compared to the stuff that didn't, even from the same authors, it's nothing). And I love using examples, too, and I sure love quotes.

But still, I think when we are this steeped in variations of the same thing over and over and over, of course we'll "consider it" at some point, and the moral or philoshopical depth is drastically reduced by already being primed. We're like people who don't see what we build with our hands, because we wear VR googles that show us movies of our childhood or some console game.

What I can see us realistically making are are "idols" with eyes that do not see, with audio output, perfect speech synthesis, that does not convey meaning, incredibly fast analysis that is not thought. From the get go, starting with the Turing test, it was more about how what something seems from the outside, than what it is to itself on the inside.

Furthermore, we might make human level AI no problem, EZ PZ, but not by making AI so smart, but my making humans dumber. We're already training ourselves to select what we consume and think from discrete pre-configured options. We notice and complain about the effects of in all sorts of smaller areas, but it's a general trend, and I think it's not so much about creating something "better" than humans, but about removing human agency.

> The frightening coincidence of the modern population explosion with the discovery of technical devices that, through automation, will make large sections of the population 'superfluous' even in terms of labor, and that, through nuclear energy, make it possible to deal with this twofold threat by the use of instruments beside which Hitler's gassing installations look like an evil child's fumbling toys, should be enough to make us tremble.

-- Hannah Arendt

Meanwhile, there's this idea that humans becoming "superfluous" means we'll all be free from "bad" work, and free for fun work and leisure. How we would get from an increasing concentration of wealth in fewer hands to some commnuist utopia? Is that some kind of integer overflow, where enough greed and power wrap over to sharing and letting others live and decide their own fate? We're connected to that (like Michael Scott is to the baby of his boss,) by delusion, the path we're on doesn't lead there.

Throw away a word here, do something that "everybody does" there, adapt to "how the world is" some, and there you go, a blank nothing that can be deprecated without guilt or resistance. The desire to control human agency is met more than halfway by our desire to shed it, to abdicate responsibility, become a piece of floatsam flowing down the river of history to the ocean of technotopia, enter the holy land of holodeck, where we can consume endlessly. We digitize, we sample, that's how we make things "manageable", and at high enough resolution we can fool ourselves, or have something "good enough to work with".

And just like children that get too much sugar too early tend to not liking fruit as much, because they're not as extremely sweet, our abstractions lure some people to prefer them over the dirty, fractal, infinite real world, or the exchange of emojis and pre-configured figures of speech over real human contact, silence that isn't awkward, thinking about what you're trying to say, or even coming up blank and that being okay... just like we go "posterized, high contrast" in all sorts of ways aready, I hve no problem supposing that we will come up with a form of alienation like that, but for thinking, I just no clue how it will look like.

We already have it with language of course, but I'm sure we can take that to the next level, maybe neural interfaces. If we can't read and transmit thoughts in their fullness and depth, then hey, just reduce our thoughts to the equivalent of grunts, that might work. Become like a computer, 0 and 1. Convince yourself that's that just what humans have been all along, remember Star Trek wisdom, don't be so proud and consider your brain more than a "meat machine", don't deny Data his quest to become human! Cue super emotional music swelling up.

That would be premise 6.

"That whereof we cannot speak, thereof we must remain silent" ~ Wittgenstein

I'm personally not terribly worried, as I think we have a ways to go before creating human-like general intelligence, let alone super intelligence. It is also not clear at all to me that an exponential increase in intelligence is likely or feasible after some threshold. Still, treading carefully seems prudent. There are some serious risks, and even if the likelihood appears low, there are a lot of unknowns.

This always puzzles me. I don't have enough knowledge about AI to be objective about that kind of statement. But deep down, I feel skeptic about it.

Not long ago I saw an episode of a show occuring in the late 80's. This kid had just received a computer as a gift and was talking to a kind-of-AI program through keyboard and screen.

The AI reactions to the kid's input was not dumber or smarter than Siri or any other currently widespread kind-of-ai program. I don't know if the show was accurate vis-a-vis this particuliar software but I like to think it was. If it was, that means in the last 30 years AI hasn't really gone further. Computer power has. Algorithms not so much.

I'd love to get the opinion of someone that has a good understanding of the current state of the AI art.

Imagine if you knew nothing about car safety features but never having been in an accident it didn't seem like cars are any safer or more dangerous than in the 60s.

The question is if we are able to get an AI to the level of intelligence where it can look at its own code and spot an opportunity to improve it, and the ability to do so, and whether or not the computational capacity immediately available to it is sufficient to reach a level where it is smart enough to find ways to obtain more resources.

The threshold for improvement can be very low if it has enough time and resources to compensate.

E.g in the most extreme case it just needs the ability to randomly flip bits and test the result, but that would require extreme resources for the initial improvements.