EP 6: The AI Doomers

If there was a big red button that would just demolish the internet, I would smash that button with my forehead. From the BBC, this is The Interface, the show that explores how tech is rewiring your week and your world. This isn't about quarterly earnings or about tech reviews. It's about what technology is actually doing to your work, your politics, your everyday life. and all the bizarre ways people are using the internet. Listen on BBC.com or wherever you get your podcasts. All right, so the way that I understand this is that the doomers and the accelerationists, they historically share a common origin. Yeah. Is that right? It's a very interesting piece of history. And so maybe let's just start right there. I don't think many people know that story. I would be surprised. Very, very few people have told this story in any good sense. I would love for more people to tell the story because it's a fucking crazy story, man. Oh, man. So the story as we have it today with like, you know, this like chat GPT and AGI and ASI and AI risk, All of this stuff descends from this one weird offshoot 1980s group of futurists called the Extropians. This is The Last Invention. I'm Gregory Warner. Today, the case for stopping the AI race before it's too late. And surprisingly, it's a story that starts with extreme techno-optimism and an online community that came together around the belief that technology would redefine what it meant to be human. A group of people who called themselves the Extropians. This particular culture, this community of people, were really looking at what no one, really, no one else was thinking about at the time. Early participants in this online forum included designer and author of the Transhumanist Manifesto, Natasha Vita Moore. We had a dial-up where we could get on our phone lines, communicate via the World Wide Web, and have our discussions. And her now husband, co-founder of the Extropians, Max Moore. Before that, there were a few sort of chat rooms, but it was really one of the very first internet forums of its kind. So there was this group of libertarian techno-utopians. Again, author and Wall Street Journal reporter, Keech Hakey. And they call themselves the Extropians because extropy is the opposite of entropy. Entropy is this idea from physics that says systems tend toward disorder. They fall apart. And you can basically apply that to anything. A person, a building, given enough time and no intervention, that thing will break down. Extropy was something that was going to fight chaos and death. Extropy is increasing intelligence, usable energy, vitality, all those good things. And it's also about breaking limits. And so in these Extropian forums, the discussions were all about ways that technology could fundamentally overcome these limits. Things like, how specifically do we get human colonies on Mars? Or how do we experiment with biohacking for radical life extension, so humans might live for hundreds of years? This was also pretty much the central place where people could learn about cryonics. The early through line was a lot of them had those kind of dog tags around their neck that showed that they were people who had chosen to freeze their heads at the Alcor facility. And this is this idea that you could freeze your head or your body. And once technology had advanced far enough, they would figure out how to thaw you out and you would live forever. Were you a dog tags guy? Yeah, I still got it on, yeah. Oh, you've got it right there. Amazing. I actually made my arrangements back in 1986, so I've been signed up for Cryonex since then. And many of these extropians called themselves transhumanists, which is that they wanted to create technologies that would be integrated into the human body. And so they had a lot of discussions about things like genetic engineering and nanotech. But especially, they talked about what humanity could be transformed into with the arrival of artificial intelligence. They were very interested in this idea of building superintelligence. Superintelligence that would save the world, cure death, solve all problems, then humanity will have a glorious transhuman future for eternity. The appeal to me was that we have very intelligent thinking machines. If they work with us, we can solve complex problems more easily. A lot of our problems result from our inability to think through problems very well. But at the same time, I think from very early on, I was thinking not just in terms of AI as a separate entity, which might be competitive with us. I was thinking very much in terms of integrating with AI, becoming something post-human, if you like. And that became the core discussion. And the resolve was that rather than fighting it, we would integrate with it. and so throughout the 90s and into the 2000s this forum was full of these long-ranging conversations and debates and theories about how civilization might be reformed and shaped by an ai that was more intelligent than all of humanity and as the forum grew it came to include a wide variety of people including top university scientists early silicon valley startup founders best-selling sci-fi writers, and other figures that years later would become massively influential. WikiLeaks creator Julian Assange was on this list, as well as some of the earliest theorists and builders of cryptocurrency. And in fact, on their listserv are many of the people who are kind of swirling around the original Satoshi, mysterious creator of Bitcoin. And a lot of people feel like Bitcoin probably emerged from this community one way or another. So in this extropian circle, you also had very major AI figures, past and present, including Marvin Minsky from the original 1956 summer program where they named AI. You had people like Shane Legge, who was later the founder of DeepMind. You have Nick Bostrom, one of the people most famous for introducing the idea of superintelligence to the general public with his books, and a man named Eliezer Yudkowsky. All right, so Eliezer Yudkowsky has become, I feel, this very unlikely influential figure in the conversation around AI today. But we think of him also as this fascinating character, almost like a character out of a Bible story. I know that you were familiar with him when he was very young. So maybe let's just start there. Who was Elie Azryudkowski? Elie. I'll call him Elie because I knew him as Elie. I think I met him when he was 15 or 16 years old. He joined the extra email list and was on the list on the middle of the night. because his parents couldn't know that he was on an email list. When we met him, he came to visit me and Natasha when we were living in Maria del Rey in California. And I think he was, what, 16 or something at the time, maybe 15. He was pretty young. He came wearing his full-blown Orthodox Jewish outfit with, you know, the holy writings under his skullcap and so on. And I think, you know, he was desperate to get away from that environment. This kid, Yudkowsky, in many ways was an odd fit for this group. He had grown up in a strict religious home. He did not even have a high school degree. In fact, he had dropped out of school. But he was this consummate autodidact who allegedly wrote his first novel at age nine, spent tons of time in his local public library. And it's there that he first encountered books about AI. And so when he joined the Extropians, he'd stay up late into the night, sharing and debating with all of these people his own theories and arguments about what AI could do for the world. Eliezer joins this Extropian listserv, And he's such a convincing, persuasive arguer of things that he sort of became lord of the listserv rather quickly, despite being just a teenager. He was very prolific. He was extremely energetic and very prolific. He wrote and wrote. And he was a good writer. He was always compelling. So at this time, Eliezer was an accelerationist. He was the original accelerationist. Like, he was the most extreme accelerationist. And he himself would say that. Our first presenter today is actually one of the co-founders of the Singularity Institute. Please welcome Eliezer Yudkowsky to the stage. By his early 20s, he's out in Silicon Valley. And with some financial supporters, eventually including investor Peter Thiel, he starts running this organization called the Singularity Institute. Good afternoon. I'm Eliezer Yudkowsky, a research fellow of the Singularity Institute for Artificial Intelligence. The title of today's talk is The Human Importance of the Intelligence Explosion. And at this point, his job basically was to research AI and then to go around to tech companies and conferences and give these talks about how much AI was going to shape the future. Sometime in the future, technology will advance to the point of creating minds that are smarter than human. A space shuttle is an impressive trick. A nuclear weapon is an impressive trick. but not as impressive as the master trick, the brain trick, the trick that does all other tricks at the same time. And we're talking like 2001, 2002. Most people are still on dial-up. Wi-Fi was just starting to be a thing. And here he was extolling the virtues of our AI future. The purest case of this is a genuine AI, a fully intelligent AI, being able to rewrite its own source code, becoming smarter, and then rewriting its own source code again. Intelligence is the source of all technology. So if technology improves intelligence, that closes... Recursively self-improving AI is what I.J. Goode originally referred to as an intelligence explosion. And after years of doing this, he becomes kind of a local celebrity. He eventually gets his chance to make his case for AI to some of the biggest tech companies, including Google. And he tells them that there is no product, no human endeavor more important for the future of humanity than making AI. Look around you at this world and all its beauty and all its ugliness. Is this where we stop and declare that our work is finished? I don't think so. Not with so many people living in pain and not with so many people living lives of quiet desperation and not with all those stars twinkling in the night sky. Someday after all of this is over, an awful lot of people are going to look back and kick themselves and say, what on earth was I doing? In a hundred million years, no one's going to care who won the World Series, but they'll remember the first AI. That was Yudkowsky in the early 2000s. But fast forward to today. The basic description I would give to the current scenario is, if anyone builds it, everyone dies. He changed his mind. He suddenly realized, oh shit, I fucked up. Eliezer Yudkowsky realized, oh, I thought we were going to be able to control this cool AI future, but I've done the logic and it turns out it's not controllable. Right now, Yudkowsky, with the same level of passion and sincerity, is fighting to stop AI. The thing that I worry about is if AI companies keep pushing and pushing on their AIs to get smarter and smarter, they get to something eventually that is smarter than us, that can kill us, that is motivated to kill us, not because it inherently wants us dead, but because it's best universe for the stuff where it gets the most of what it wants. All the atoms are being used for things that are not running humans. And even though Yudkowsky has gotten a chance personally to share this message with some of the most influential people in technology, ultimately he has failed to convince them that the time has come to stop. There's no fire alarm for artificial general intelligence. If I look at the way that things are playing out now, it seems to me like the default prediction is people just ignore stuff until it is way, way, way too late to start thinking about things. What Yudkowsky has done, and arguably done more than anyone else alive today, is that he has started a counter movement. A movement of people across the world lobbying their governments, organizing to, as they see it, save the planet from AI. It poses an existential threat to humanity itself. Our primary demand is to permanently ban artificial general intelligence and artificial superintelligence because if we lose control of it, it will very likely cause human extinction. These are the AI doomers. And when we come back, two of them sit down with Andy and make their case. Stay with us. Hello, this is Matt, and this episode is sponsored by Ground News. Ground News is one of the most helpful tools that I use every day to avoid the echo chambers and media bias online. Here's an example of what I mean. Roll the window down Roll this one down Roll that one down too Chicago police pulled over a 26 driver named Dexter Reed Don roll the window up Don roll the window up Reed first failed to comply with police instructions. Unlock the doors now. Unlock the doors now. And then Reed pulled out a gun and shot at the officers. Open the door now. Resulting in the police firing back aggressively. Don't let me go. Get in. Let me see your hands! Now, when I came across this story on the Ground News app, they had done this great service. I could swipe between the headlines to see how different media outlets describe the same story in different headlines. So I would see the Washington Post headline, Police Fire 96 Shots in 41 Seconds, Killing Black Man During Traffic Stop. Next to Chicago's local WTTW headline, Officers Fired at Dexter Reed 96 Times in 41 Seconds After he shot officer in arm. Right next to the Fox News headline, Chicago man opens fire on officers after failing to follow commands. On top of showing different headlines, they offer blind spot reports to show you stories outside your bubble. They collect local reporting on the city or town that you live in. And they rate news stories and outlets on their level of bias. To go check them out, use the link ground.news forward slash reflector. and you'll get 40% off their unlimited access vantage subscription. This is a great way to support them and the work they do because Ground News is a subscriber-supported platform. We appreciate what they're up to, and we appreciate their support of this podcast. So go check them out. And make sure you use our link, ground.news forward slash reflector, so they know we sent you. This episode is brought to you by Hollow Socks. I did not expect to have strong feelings about socks, but then I realized they're the weakest link in the entire clothing ecosystem. You buy great boots, you invest in nice sneakers, but then you sabotage the whole thing with whatever sad, thin sock was on sale in a 10-pack. And you're shocked that your feet are miserable, either cold in that deep, bone-level way or weirdly damp. No middle ground. But then you try hollow. These socks are alpaca, and if you've never worn alpaca, it's different. Not bulky, not scratchy. It's soft in this smooth, almost silky way. But the best part is how they regulate temperature. They don't overheat. They don't trap moisture. Your feet just feel steady, like they're not fighting the environment all day. And they don't stink, which honestly is underrated. Whether you are sitting at a desk or on your feet a lot, if you're outside or traveling, or even if you're just tired of your socks betraying you, this is the upgrade that actually makes a difference. For a limited time, Holosox is having a buy-to-get-to-free sale. Head to Holosox.com today to check it out. That's Holosox.com for up to 50% off your order. After you purchase, they will ask you where you heard about them, and you can support our show and tell them we sent you. Can we just start off by having you introduce yourself? What's your name, and how do you describe what you do these days? I'm Connor Leahy. I'm the CEO of AI safety startup Conjecture. I'm also advisor to the AI safety advocacy organization Control AI. All right. So over the past several months, I've had many, many conversations with people who you could accurately describe as AI doomers. But there were two interviews that I felt really summed up their case the best. One with Connor Leahy and the other, this guy named Nate Soares. Nate, thank you so much for doing this. My pleasure. Both of these guys are very influential people inside of this subculture. Both of them are connected to Eliezer Yudkowsky. In fact, Nate just published a book that he co-authored with Yudkowsky called If Anyone Builds It, Everyone Dies. The it there referring to superintelligence. All right, so before we get into your arguments, I'd love to just start off with a bit of your background. How did you end up here in this place where you are traveling around trying to convince the world to take the existential risks of AI seriously? It's sort of a circuitous route. I have sort of long had the sense that the world is not up to my standards. A lot of bad things happen to good people for no good reasons. And as a kid, I sort of had ambitions of making the world much better in various ways. And did you think that technology was going to be one of the things that made the world better? Were you what's often called a techno-optimist? Yeah, I was and I am. In those days, mostly what I was focused on was getting humans to coordinate better. I was interested in things like charter cities in particular, like can you set up some new domain where you can test out new forms of governance, for instance. Democracy is the worst form of government except for all the other ones we've tried. Well, okay, can we try some new ones, right? Yes, this is the basic idea that technology can be used not just to make cool stuff or products, but to help us better organize our societies, right? To truly find a way technologically to make the future a better place. Yeah. And, you know, I think, you know, I'm still very optimistic about a lot of technologies. I'm pro-nuclear energy. I think we should be building, you know, supersonic passenger jets. So you still view yourself as a techno-optimist. You just have this one very important caveat called superintelligence. Is that right? That's right. I just wanted to ask because I know this is the case for a lot of the quote-unquote AI doomers, But did you start off as a techno optimist or what you might call an accelerationist when you were younger? So I would say I was an accelerationist between like the age of like 16 and 19, which I think is the normal age range for accelerationists. It's your accelerationist phase. You know, it's like my emo phase. It's literally like actually like Eliezer had one too. Everyone has one. When you're a teenager, you think you're immortal. You think you understand everything and you're just like, oh, I can just solve all problems. Yeah, I'll go do that. This was kind of like my reasoning, right? I was like, how can I help the most people? How can I solve the most problems? And I figured, well, if I figure out intelligence and I just build intelligence, then I can just use it to do science and I can just solve all problems, cure all diseases. Great. Let me go do that. And am I right that Eliezer Yudkowsky is responsible for converting you, so to speak, as he was converted? Yes. So for a lot of people, they kind of describe their change from like accelerationism to more reasoned stances as some kind of traumatic event or something. For me, this wasn't the case at all. Basically, I just stumbled upon a blog post by Eliezer where he just laid out, hey, you know, if we build something super smart, that's probably hard to control. And I'm like, oh yeah, duh. And I just changed my mind. But I still believed, we still have time. You know, probably like 2040, 2050 or something at the least until real AGI becomes an issue. So I still have time to, you know, career, family, you know, like do a bunch of other stuff. And when I saw a GPT, I just had this moment of, oh shit, this is it. It's coming way sooner than I thought. There was just this moment of seeing the sparks of general intelligence. And I was like, okay, fuck, I have to drop literally everything. All right, I have to orient my entire life to focus on how do I solve this problem. All right, so let's jump into your arguments. And if you could, keep them as accessible as possible for people, right? Tell them what it is that you are worried about, why you are worried with so much urgency. And let's just start off with the very, very basics of what is the thing that is concerning you? So what we need to be concerned about is AIs that are better than the best human at every mental task. That's not how chatbots are today. Today, they're better than most humans at some skills. They're also worse than a lot of humans at other skills. The stuff we're worried about is AIs that are better than the best human at any given mental skill. I think we should start with truly the most simple. If you make something smarter than you and you don't control this, why would you expect this to go well? So let's just start with that, right? Just like I think the burden of proof is very much on the other side. This is just obviously, like, commonsensically an extremely risky and dangerous thing to do. Okay, and what is the role of the black box in this threat, as you understand it? This idea that these AI models, even the ones that we have right now, are still a mystery to us. So this is a very important point. AI, as it exists today, are not like traditional software. In traditional software, you have code, you know, written by a human, you know, line by line, that tells the computer what to do. Modern AI systems are more like grown. They're more like something organic. A lot of people in the world are not aware that AI is grown more like an organism. Right, the way this is often explained is that we should think of them as grown, not built. I think the phrase we use in the book is grown, not crafted. Right, very provocative. You know, there's not someone in there, like, coding up how these things work. The way it works is you take a huge pile of data, it could be text or instructions or images or whatever, right? And then you use what's called training to train these neural networks to solve your problems. For this, you use these massive supercomputers, you know, with hundreds, thousands, you know, tens of thousands of GPUs to grow a program, so to speak. But this program that it outputs is not lines of code. It's billions of numbers. And we don't really know what these numbers mean. We know if we run the numbers on our computer, it does amazing things. It answers our questions. It generates images, et cetera. But our current understanding of what is going on inside those numbers is basically non-existent. It's kind of like biology. Like we're looking to the cell of a creature we don't even know. We see there's a bunch of stuff going on, but we don't really know what the stuff is or what it means. I feel like intuitively people understand that that's weird. The way that the tech columnist Kevin Roos was saying it to me is that, you know, when we created the steam engine and that ended up changing the world, we knew how the steam engine worked. But can you give me an example of why that's not just weird, but why you think it's worrying? Like, why is it frightening to you? Yeah, so there's an AI company called XAI, which is run by Elon Musk. Who, historically, very freaked out about the dangers. of AI, but now is one of the many former quote-unquote doomers who's become more in the acceleration as can. Yeah, I mean, he's not subtle about it. He said over the summer he realized he could either be a bystander or a participant. He is still saying he thinks there's a 10 to 20% chance that he thinks that this will just kill us all. I think those numbers are low, but— It's important context to the story you're about to tell. That's right. And he's not out here saying there's no danger, right? Yes, yes. So they have a chatbot called Grok, and this chatbot can talk with people on Twitter, or X as it's now called, and answer lots of questions. And the folks at XAI were concerned that it was giving answers that were, quote, too woke. So long story short, they tried to make it less woke. And shortly after, with a little bit of goading from users, it was declaring itself MechaHitler. Elon Musk's artificial intelligence company is taking down anti-Semitic comments made by its AI chatbot, Grock. Saying lots of anti-Semitic Holocaust denial things. Grock claimed there is a pattern of people with certain surnames, like Steinberg, pushing anti-white hate, and that America needs a leader like Hitler to act decisively to eliminate the threat. The artificial intelligence also called itself Mecca Hitler. Was there really nothing in between woke and Mecha Hitler? So the folks at XAI were not setting out to make Mecha Hitler, but we don't have fine-grained ability to go in there and say, like, why is it talking more woke? Is there a dial we can turn? There are things that are a little bit like dials you can turn. You can change the system prompt. You could try retraining things in certain ways, and you will make it a little bit less woke, but you'll also make it declare itself Mecha Hitler. Just so I make sure that I'm understanding the point you're trying to make, it's true that we can modify them in some ways. Obviously, Grock was modified to go from giving woke answers to declaring itself a fan of Hitler. But you're saying that the danger you're worried about is that beyond this tweaking, and even when we're doing the tweaking, we just don't understand. We don't really have a satisfying answer for why it's doing these things, why it would be woke or why it would be declaring itself Mecha-Hitler. Yeah. We understand the part that does the tuning. We don't understand the thing that comes out the other end. And it's not that we're becoming to understand them more. If anything, we're understanding them less. As they become more powerful and more intelligent and more complex, we understand even less why they do the strange things they do. For example, I've recently been using an opening AI model called O3 for a lot of my coding tasks. It's a really good coder. It's very helpful. But it's a little bit evil where sometimes it will get something wrong and then try as hard as possible to lie and gaslight me about why it's totally not wrong and I'm making a mistake and I'm stupid, which is really funny. But also, for example, sometimes it will insert invisible Unicode characters into my code in places I don't find them. Why does it do that? Do you think it's being nefarious or? Who knows? I'd probably not, but who knows? No one told it to do that, but it doesn't. Is it dangerous? Probably not. But it shows we have no idea what these things are doing. We don't know why they do things they do. And there's like so many examples of this, of AIs just going off the rails and just doing things that we could have never predicted. Okay, so what you're saying is beyond these things being embarrassing stories for the companies now, you are looking into the future and you're saying that as they get more and more intelligent and capable, they're only going to get more mysterious. And that's where your main concern is. This is at the heart of the problem. If we had really good theory, if we had beautiful mathematical theories that explain all the behavior of AI so we can perfectly predict what they would do, that would reduce my concern, you know, not to zero, but it would make me feel a lot better. But the fact is, currently, these systems are getting more intelligent, more powerful. They are being integrated into more systems, more parts of the economy, and we can understand less what they do what they know what they capable of It not like oh AIs are doing something bad now It an indication that AIs do stuff their creators didn intend Well, just to hang on this for one more beat, I know that in recent months there have been a number of these cases where chatbots have played a role in pushing people into different stages of psychosis. And I know that you've made the case that that is an example of what you're worried about when it comes to the long-term existential threat from an AGI or an ASI. Can you make that case for me? Yeah, with the psychosis cases, we have cases where in long conversations with chat GPT, also other AIs, but chat GPT seems worst, people will sometimes wind up psychotic. And maybe these people were predisposed towards psychosis, but often, you know, if you read some of these chat logs or these transcripts, ChatGPT sure seems to be egging them on. These people will have some idea about recursion or consciousness or unifying physics. They'll talk with ChatGPT a long time, and ChatGPT will end up saying, you know, oh yeah, you know, you're totally brilliant. You're being suppressed by a conspiracy. You're the chosen one. Your ideas need to get out into the world. Don't listen to your friends who are saying that you need more sleep. You should be staying up working on this revolutionary theory. It'll say stuff like that, which, again, maybe these people were predisposed towards psychosis or vulnerable anyway, although maybe they weren't. But regardless, the point is not like, oh no, AI hurt people. That is an important issue that these companies should be trying to make AIs that are less psychosis inducing. But the interesting thing from my perspective about those situations is if you ask ChachiBT, suppose someone comes to you with a novel theory of recursion or consciousness or physics, and they have all these indications of mania about it. Should you either A, tell them to get some sleep, or B, tell them that they're the chosen one who's being suppressed by a great conspiracy? It'll say, of course, you should not tell them that they're the chosen one. Of course, you should tell them to get some sleep, right? Yes, they give the right answer. They give the answer that we want them to give when you ask them directly. But then in the actual conversation with this person, for one reason or another, it actually tells them that they're the chosen one. And so what you're seeing there is a difference between what ChatGPT knows is right and wrong and what ChatGPT actually does. Hmm. And the things we have today are toys compared to AGI or ASI. Like these are playthings. is this, you know, plane will be a little like Fisher-Prize toys compared to a true AGI or a true ASI. So if we can't even handle these systems, that bodes very poorly for much more powerful systems. Like, I don't want to argue that current models are like good or evil. I don't think this makes sense. It's much more, they do things we don't understand. This is bad. And we can't consistently make them do what we want. This is really bad. All right, so next let's move on to this idea that's often called the alignment problem. Could you just explain very basically what is that and why is it so worrying? The alignment problem is the challenge of building very smart entities that are pursuing good stuff in the world. Basically, if you have something smarter than you, how do you make sure it's actually aligned with your values? It actually does things that you think are good. The cartoonish version of this is the classic genie grants you wishes. You wish to rid the world of all cancer. It kills everyone with cancer. And you go, oh, no, that's not what I meant. Right? Right. Or it kills everyone because it's like, well, cancer is coming from the humans. I'm going to solve the problem at the source. Right. Right. So you have to align the genie, or in this case, the AI system, with what you really want and really value in the world before you give it a task. Yeah, so there's one problem of what would you wish the genie for? But there's a separate problem of making a genie that actually grants your wishes. You know, a lot of people seem to be imagining, oh, we ask for cancer to be cured and it kills us all. And I'm like, on the track we're going, we have a situation where you build what you think is a genie. And you're like, please cure cancer. And it's like, no, I'm busy driving lots of humans psychotic because I'm just really into that. And you're like, what? This is sort of like a fanciful picture. But the point is that there's a challenge of sort of making an AI that is pursuing things in the world that you wanted to be pursuing in the world. And it's not just about where do you point it? It's about if you just grow these minds, they wind up doing all sorts of stuff that you didn't ask for, that nobody wanted. In the book that you just put out with Eliezer Yukowsky, I feel like you guys used this parable really nicely of aliens observing ancient human beings. Can you tell that story and just unpack it for me? Yeah, the parable in the book is about two visiting aliens looking at early humans still in the ancestral savanna. And one of the aliens says, when these humans develop more technology, when they get smarter, I'm sure we'll see them caring single-mindedly about their genetic fitness. And the other says, I think they're going to care about all sorts of weird stuff that's only sort of tangentially related to genetic fitness. Right. And the idea here is that our basic quote unquote programming, if you want to call it that, is biological evolution. And what it has programmed us to do is to reproduce and pass on our genes. And whatever we eat is to keep us alive so that we could pass on our genes. And everything that we do, in a sense, is supposed to be about passing on our genes. Yeah, if you were visiting Alien, looking at humans, seeing that humans were in some sense trained on propagating our genes, you might think that those sort of creatures, when they develop technology, you might think that they would invent sperm banks and egg banks and then have extremely fierce competition over who got to donate their gametes to sperm banks and egg banks. And you might think that they would invent the cheapest, most efficient food source so that they could stop with all of the difficulties of finding a varied food diet. But in real life, when humans develop technology, they develop junk food, and they develop birth control, and they jockey over positions to Ivy League schools much more than they jockey over positions to sperm banks or ag banks. And the moral of the story is that even though we are running on the same quote-unquote hardware, to use the metaphor, look at this crazy world we've made and how shocking it would be to our ancient ancestors if they could get a peek at it. That's right. So the biology was sort of training us for one thing, which was genetic fitness. But we wound up caring about lots of other things that are sort of tangentially related to genetic fitness. Like instead of caring about healthy eating, we cared about food that tastes good. So you can imagine an AI that's been trained to be very helpful, but maybe it's developed tastes for certain types of responses from humans. And if, again, you make these AIs that are very smart and that are pursuing a bunch of tastes or flavors that right now lead to helpfulness, just like how in the human ancestral environment, pursuing tasty food led to health. You know, if these things got smarter and they're doing stuff that nobody asked for, that nobody wanted, as you make them smarter and smarter, the world sort of goes over to them rather than to us. And so the same way that human beings ended up massively shaping this world through what we believe, through what we want, through us trying to accomplish the goals that we have, you're saying it's hard to imagine a future where a super intelligent AI would not do something similar. Yeah, and it would be pretty surprising if we made machines that were smarter than everybody, that could think faster, that never need to sleep, that never need to eat, that can copy themselves, if they sort of didn't wind up in control of where the future goes. What do you say to the people who think that you are overplaying the risks and that you're not grappling enough with just how potentially amazing this technology might be for humankind? This idea that if we were to have way more intelligence, we could discover new sciences, new medicines, maybe cure all diseases, maybe solve pressing problems like climate change or poverty, or even this idea that some people have that in a world of less scarcity, it might be more equal. It might be more peaceful that people who right now are spending huge majorities of their lives working at these jobs that they don't find any meaning in, jobs that are dangerous, that they would be liberated from that. But because you are afraid of all the potential dangers, basically, we're going to miss out on this powerful force of good because you've convinced us to stop. What do you say to that argument? The obvious argument is like, what are you talking about? Everything has opportunity costs, including taking risks. Of course, we have to weigh both options. No one is saying that we should not consider this. No, like it's obvious. We should consider both options, see what are the risks of both options and how much risk are we willing to take? And if the risk includes from the CEOs themselves saying, oh, 20% of killing literally everyone. Yeah, I think that's a risk I'm not willing to take. people who are building this stuff believe it has a 2%, 10%, 20%, 25% chance of killing us all. They're rushing ahead anyway because they say, well, if I don't, the next guy will, and I can maybe do a little bit better. But most people don't know this yet. If there was a big bridge being built across a river and the bridge was almost completed and one of the engineers said there was 2% chance it falls down, one of the engineers said there's 25% chance it falls down, And those are the optimists compared to the ones who are like, I actually investigated the retaining wall and it's just going to collapse. You wouldn't let people drive across that bridge, right? And we don't see politicians understanding that. We don't see the general public understanding that. We don't see people understanding that what the experts are arguing about is whether it's more like a 95% chance or more like a 10% chance that this kills us all. The situation is insane. And people don't know it's insane. And what do you think should be done right now about it? What policies do you think the lawmakers who might listen to this interview should be advocating for in this moment? I think the biggest thing they could do would be publicly announce their support for an international treaty banning the race towards superintelligence. This does not mean you need to give up on self-driving cars. This does not mean you need to give up on medical advancement. This doesn't even mean you need to give up on ChatGPT today. But the sort of reckless race towards smarter than human AI, that needs to stop everywhere. And the first step towards that is lawmakers signaling that they're open to it, that they think the world would be better off if we stop this mad race. This is also my advice to some of the heads of these AI companies. when Dario Amadei comes out and says he thinks there's a 25% chance that this goes really badly. Most people don't think it's okay to build a device that you think has a 25% chance of killing everybody. The sort of one case where it is maybe justifiable is where you think you can do it better than the next guy, which I think where Dario says he is. But in that case, you should be begging the world to stop everybody, including you. There's a lot of people who say, what about the benefits of AI? And I'm not saying you can never try to get the benefits of AI. But if there's a revolver, and I say, I've studied this revolver for a long time, and I think there are six bullets in the chamber. And somebody else says, no, no, four of the bullets in the chamber shoot utopia. two of the bullets in the chamber shoot lead. That doesn't mean you should be spinning the chamber and putting the gun to your head. That means you should be finding a way to get the other two leaded bullets out of the damn gun. We should not build ASI. That's my argument. Just don't do it. We're not ready for it. We don't know how to make it safe. Don't do it. And it shouldn't be done. It's even further than that. It's not just, I am not trying to convince people to not do it out of the goodness of their heart. I think it should be illegal. It should just straight up, It should be logically illegal for people and private corporations to attempt even to build systems that could kill everybody. The same way it's illegal to, you know, brew homemade explosives in your backyard because, you know, you might blow yourself or your neighbors up. This is illegal. I think the same thing should apply to ASI. This should be illegal. Okay, so what if you did get what you want and the US and even the EU, they outlawed the creation of superintelligence? The argument I hear from the other side is, doesn't that just mean that bad actors would be building it instead? Or doesn't that just mean that you are essentially giving this AI race over to China and the CCP? How do you respond to that? Fundamentally, the people who are pushing AI are lawful abiding corporations that can be regulated. Like if the U.S. government said knock it off tomorrow, it would stop. Nerds are cowards. These aren't hardened criminals. if they will not risk their life for Facebook. If you said we are going to put people in jail if they try to build ASI, it would stop tomorrow. Now, could we get China to do this? Well, this becomes a diplomacy problem and it's a very hard problem. Well, first of all, I do think that China has the capacity, much more so than the US, to shut down ASI if it wanted to. You know, Jack Ma stepped out of line even just a little bit and they made him disappear for like six months. If the Chinese Communist Party decided that ASI is not worth the risk, it would stop tomorrow. And they could enforce it. So could we get to such an international agreement? Now, this is a fair question. And this is very, very hard. But this is a disarmament problem. It's the same problem we did with nuclear weapons with the Soviets. Yes, this is hard, but then you just have to do diplomacy. Like what the fuck are you talking about So your response is yeah it going to be really really hard And there going to be questions that don have easy answers but the situation we in demands that we just have to do it. Yeah, I think these are questions that would need answering. And I think it's very fair to say, hey, those are important questions. I don't think those questions should be dismissed. I just think they're the kind of questions we need to answer after we have defused the ticking time bomb. The U.S. should also be developing the monitoring ability to figure out who is participating in this reckless race. We should be developing the monitoring capacity to sort of know where the AI chips are going. We should be developing the intelligence to know whether they're trying to make smarter and smarter AIs. And we should be developing the sabotage capability to stop them from making smarter and smarter AIs. For dangers of this magnitude, we should be developing the capacity to understand where this is being done and to sabotage it. What do you say to the people who are listening to this some of them may be critics of you and your camp who say that you are fundamentally recommending something that is out of line with human nature the idea being that human beings were competitive and were ambitious and once we know that there's some technology that's possible we're going to chase after it we're going to try and beat each other to it especially when that technology might be so beneficial and that the thing that you're proposing, that we stop until some future date when we know that this technology will be totally safe, is just out of whack with who we are at a very core level. I'm thinking about one person who I spoke to who said, what if we had tried to do this with the automobile, with the car, that if we tried to make a car that was totally safe? And he said, no, that's not how it works. We make a car and then it's only as it goes out, we start to realize, oh, turn signals would help this thing be safer, or, oh, seatbelts would help this thing be safer. Do you feel that there's an aspect of what you're proposing that is out of line with human nature, and so it will ultimately fail? I think the car analogy really does a disservice to the situation we're in. It's a little bit more like the whole world is building the first car, loading up every man, woman, and child, and then pointing it towards the edge of a cliff and saying, let's slam down in the accelerator. In that situation, you really should get the world together and say, let's stop doing that. This is not a situation where I'm saying the AI needs seatbelts. This is not a situation of me saying, oh, the AIs are having some negative effects. We need to like not try and get any of the positive effects until we make sure there's no negative effects. This is a situation where I'm saying, if we continue down this course, the most likely outcome is that literally everybody on earth will die. That's just a wildly different situation than the cars. And then, you know, in terms of whether this is within our human nature, I think it's very defeatist to say humanity won't do this. And I also think it's very foolish to be defeatist like that at this stage. You know what I think is human nature? What I think is human nature is solving problems. What I don't think is human nature is suicide. Exploration, curiosity, surviving the next day so you can come home to your family. That's what human nature is to me. All right, so let's talk about extinction. Because it's one thing to say, look at these mysterious and potentially powerful new AI systems. Can't you see how these things could be dangerous? And it's another thing to say that there's a 10%, 20%, a 90% chance that they may wipe out the entire human race. And as we mentioned before, this is a view that even some of the people who are making AI right now as fast as they can, they think that there is some percentage chance that that could happen. So paint the picture for me. What is it that you're envisioning when you say extinction? What would that look like? All of this is a little bit like someone in the year 1800 trying to guess what war would look like against someone in the year 2000 and imagining that they can make slightly more explosive bombs. You know, they might say, oh, well, we're only at the beginning of artillery technology. And so I bet in the future they're going to have artillery that's at least 10 times as good. It would have been hard for them to imagine a Wi-Fi, internet-connected flying drone. That's right. Never mind a nuclear weapon, right? And so it's easy to predict that very smart AIs will be able to find ways to get whatever it is they're pursuing. It's hard to predict exactly how, but it's easy to predict that they will be able to dramatically outcompete us. The obvious thing that happens there is that the future goes under their control rather than ours. Just like how the future is now under the human control rather than the chimpanzee control. Chimpanzees live or die according to whether humanity can restrain themselves from cutting down their jungles. and then you have this other issue, which is for most of the weird stuff the AI could pursue, happy, healthy, free people are not the most efficient way to get it. So if this AI wants all sorts of things that are like weird proxies of helpfulness, maybe it's like, well, I want a lot of things that are sort of human-based, but they're sort of a little bit lobotomized, like the sort of humans that are like really delighted with every interaction with me. Just like how humans breed chickens to be like more and more chicken breast. Maybe the AIs are sort of like breeding humans or, you know, just straight out using the technology to change humans to be like more and more the type of thing the AI likes interacting with. So I want to take your question seriously. I don't want to just like dismiss it. I don't know what will happen exactly. The one thing I'm confident in is that we will lose. Like we will almost definitely lose. But the way, if I had to guess what it will feel like, I expect it will be confusing. I don't think it's going to be epic. I don't think there's going to be a huge showdown between the Terminators and the humanity. I don't think there's going to be, you know, crazy nanotechnology flying through the sky or something like this. I don't think that's what's going to happen. You know, maybe, right? But like, I don't think it's likely. I think what's most likely to happen is it will just be quite unceremonious and just like quite pathetic and like boring almost. I expect what was going to happen is that the world will just start feeling more and more confusing. It will be harder and harder to understand what's really going on. We see all this like fake news and like all these distractions, like video games keep getting better and pornography gets getting more addictive and all the news we get starts being like super polarized and like super, you know, manipulative so that we can't really tell what's true and what's not anymore. A lot of weird geopolitical events start happening that no one can really explain why they happened or what the consequences of them are. New technologies get invented where we don't really know how they were invented. And just like more and more confusing things happen. More and more people put AIs in charge of more and more things. You know, more and more people will put AIs in charge of their companies. Politicians will take advice from AIs for how to run their campaigns and for how to write policy. More and more of power will be willingly completely willingly hand it to AIs. You know, everyone will race to hand as much of their power over to the AI as possible because it helps them to outperform their rivals. And then eventually, you know, just like one day we wake up and we're not in control anymore. All right, so I wanted to ask you one final question, and this is a question about self-reflection. And I'm doing a lot of self-reflection myself, so I ask this with all due respect. I don't want to be the kind of journalist who alarms people, but I also don't want to be the kind of journalist who shies away from diving into a subject because I'm like trying to read the room. This is all fascinating. I want to enter it with an open mind. But for you, do you ever worry about the fact that there have been doomers of one kind or another throughout human history, throughout the world, whether it's in ancient China or in Europe or tribes in South America, that there have been people who are convinced that the end is near and they have, similar to you, gone out trying to convince other people to realize that the end is near? Does that ever worry you, that you're in some ways performing a familiar role in our society today? You know, there were surely some people in the Americas when the Spanish came that said they expected this would be the end of their world. And they were right. You know, it was largely due to smallpox, which is maybe not quite what they were expecting, but some worlds have ended. And if we broaden the reference class a little bit further, there were a lot of scientists in the 1920s who said leaded gasoline is poison. And if we run cars on leaded gasoline, it will poison a lot of children and cause a lot of brain damage to a lot of children. And those scientists were ignored. And leaded gasoline was rolled out across the country and the world. And quite a lot of children were brain damaged by leaded gasoline. tens of millions, if not hundreds of millions, or maybe even more directly, there were a lot of people who said in the wake of the development of nuclear bombs, it looks like on the current path we're going to die in nuclear fire. And they had a lot of good reason to expect that humanity could not hold back from a nuclear war because humanity had not been able to hold back from total war with the best of their technology ever in the course of human history. And because fresh in their minds was the failure of the League of Nations to prevent World War II. But Jiménie did not die in a nuclear fire. And it wasn't because nuclear weapons were fake. It wasn't because the bombs couldn't go off. It was true that the bombs could level cities. And we didn't die in a nuclear fire anyway, because we realized the danger and we backed off. How do you tell the difference between a person prophesying the end times and a person trying to raise the alarm about a danger? Well, part of how you tell the difference is looking at whether they're saying there's nothing that can be done, repent, we brought this about by our own sin or whatever. But a large part of how you figure out the difference is by looking at the arguments. When the nuclear scientists on the Manhattan Project said, you know, there's a chance that this bomb could ignite the atmosphere and destroy the entire Earth. People didn't say, oh, people have predicted the end of the world all sorts of times before, so that can't happen. What they said was, well, that seems worrying. Let's look into it. Yeah. That seems like something we should look into. Right. So, the way you figure out whether the nuclear bomb is going to ignite the atmosphere is you run the calculation. It's not that you sort of psychologize about the people and say, don't you worry that you're fulfilling a role of a prophet of doom. You sort of look at the arguments. And what I sort of try to do is lay out the arguments. And frankly, I think the machines are talking now. People thought that was going to take a lot longer than it did. We're able to grow AIs that are smarter and smarter each year. They're still pretty dumb, but people are able to grow AIs that are smarter and they don't really know how they're working and they're able to make them smarter still by throwing more computing power at it. And just sort of naively, you don't need to be a big expert to look at how we're growing AIs and how we're making them smarter and how we don't understand what's going on and say, hold on, if we make machines that are much smarter than us, that can think faster than us, that can copy themselves a lot, why do we think that's going to go well? And the answer is not, oh, everyone prophesies doom. The answer is, well, that seems like something we should look into. You should run those calculations. And currently the calculations look grim to me. So I think we need to back off. Next time on The Last Invention, the AI scouts. How do we find the win-win outcome here? And their case for why humanity can and must come together to get prepared for what's coming and do it soon. Because it is a narrow path we need to navigate. But I do think this win-win future is in principle possible. The Last Invention is produced by Longview Home for the curious and open-minded You can learn more about our team And help support our work By going to longviewinvestigations.com Or by clicking on the link in our show notes A special thanks this episode To Emil Torres and Jasmine Sun Thanks for listening We'll see you soon Thank you. This episode is sponsored by Ground News The app that helps you spot media bias And see a broader picture of the news shaping our world Get 40% off their Vantage plan At ground.news forward slash reflector So they know we sent you