AI Wants to Be Your Friend. That’s the Problem.

This is an iHeart Podcast. Guaranteed human. Run a business and not thinking about podcasting? Think again. More Americans listen to podcasts, then add supported streaming music from Spotify and Pandora. And as the number one podcaster, iHeart's twice as large as the next two combined. Learn how podcasting can help your business. Call 844-844-iHeart. On the Look Back at It podcast. 1979, that was a big moment for me. 84 was big to me. I'm Sam Jay. And I'm Alex English. Each episode, we pick a year, unpack what went down, and try to make sense of how we survived it with our friends, fellow comedians, and favorite authors. Like Mark Lamont Hill on the 80s. 84 was a wild year. I mean, it was a wild year. It was a wild year. I don't think there's a more important year for black people. Listen to Look Back At It on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Imagine an Olympics where doping is not only legal, but encouraged. It's the enhanced games. Some call it grotesque. Others say it's unleashing human potential. Either way, the podcast Superhuman documented it all, embedded in the games and with the athletes for a full year. Within probably 10 days, I'd put on 10 pounds. I was having trouble stopping the muscle growth. Listen to Superhuman on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Hey, this is Robert from the Stuff to Blow Your Mind podcast. Joe and I are both lifelong Star Wars fans, And so we're celebrating May the 4th with a brand new week of fun, thought-provoking Star Wars-related episodes. Join us as we tackle science and culture topics from a galaxy far, far away, such as the biology of tauntauns and wampas on the ice planet Hoth, or the practicality and corporate business sense of the Sith Rule of Two. Listen to Stuff to Blow Your Mind on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. I think the key principle is, is the technology being designed to replace teachers, or help make teachers better teachers? Is the technology being designed to replace your relationships or deepen your ability to have better human relationships? Tristan Harris loves technology. He wants to live in a world where tech is in the service of people. It fosters human growth and connection. Unfortunately, right now, that's not exactly what tech is doing. And Tristan and his nonprofit, the Center for Humane Technology, are trying to help. Tristan's background is really interesting. He's a Bay Area kid, educated at Stanford with all the other tech leaders. His friends in college actually started Instagram. He launched his own startup that was acquired by Google, where he went and worked for a number of years. So Tristan knows and has known the handful of people who are designing and building the technology we use every day. Technology like Instagram and other social media apps that we're all now addicted to. Through the Center for Humane Technology, Tristan has been raising the red flag about social media, pointing out the many ways that these platforms exploit human psychology and vulnerability, how they can isolate us and make us dependent on the doom scroll. It's a model that has led to what Tristan calls human downgrading, where the tech gets better, but it changes our lives for the worse. And it's a model that AI is now following. Tristan is concerned that AI has potential to be the next social media. Only this go around, instead of hacking our attention, AI is hacking our attachment. I'm Lori Siegel, and you're listening to Mostly Human, a tech podcast through a human lens. Tristan, I've known you throughout the years, and you've always been the person who's speaking very eloquently about what's coming around the corners and what we're not talking about. So let's go to this moment. You were talking about social media for the last like 15 years and the harms if we're not careful. But now we're in an AI moment. And this moment, I almost feel like, and you can correct me if I'm wrong, the stakes are even higher. Ground us in this new reality. So, you know, probably when you and I first started talking or maybe at first introduced to each other, in 2013, how could we predict so much of the effects that social media would do? A more addicted, distracted, polarized, sexualized culture. we could predict all of that because of one thing, is if you look at the incentives, meaning that people say, well, social media, it's giving people a voice, helps people connect with your friends. Yeah, it does those things. But is Facebook's business model, is TikTok's business model improving the health of society or giving people a voice? Or is their business model showing whatever at your nervous system keeps you scrolling in a doom scrolling loop? And obviously it's the latter. And so I think one of the key things we have to do is demystify which direction technology is going by getting clear about what the incentives are. So if you flash to AI, there's all these things it could do. Why are we seeing the major AI companies release these AI deepfake slop apps, meaning like it's like a TikTok, but it's just AI generated deepfake slop. So Meta released Vibes, OpenAI released Sora. And why are they doing that when they said they're here to cure cancer and they're here to solve climate change and they're releasing something that just keeps people scrolling? And the answer is because they're racing for market dominance. They're racing to get to AGI. They're racing to get to basically owning the world economy first. And what's important about getting that is that we are not going to get this utopian world on the current trajectory because they're racing to release the most powerful, inscrutable, uncontrollable technology we've ever invented with the worst possible incentives of getting there first rather than making sure we do those things right. You talk about the incentives, right? You talk about the incentives of this new generation with artificial intelligence, same incentives for social media. Well, yeah, there's so there's different things here. So what was behind social media was the attention economy. There's a finite supply of human attention. The companies are in an arms race to harvest bigger and bigger slices of it. When they do that, they have to aggressively push out the other guy by going deeper down into your brainstem, hacking social psychology, hacking fear of missing out, hacking social validation. And that's what got us the catastrophe that we're now living in. And so when you watch one huge societal catastrophe unfold through a kind of blind spot of not facing the consequences of what's really at stake, having gone through that experience, it's like, I don't want to see that happen again with AI. I think we both probably feel this so personally. Yeah. I think about, you talk about like, you went to college with these folks. Yeah. I think about being a young journalist, 23 years old in the CNN newsroom and being like, there's this really cool thing happening. It's called tech. And I remember always thinking like and asking these human questions of like, oh, like, that sounds amazing. But have you thought about this? And I think this is why this moment is really personal to both of us, because you, your background was at Google and in design, and you saw how people were actually designing products to make us addicted to some degree. And so now here we are. It's 2026. You know, it's such an extraordinary moment for us to try to get this right. And I go back to 2024, October. Your team at CHT, and this is the stuff that you focus on, introduced us to a woman named Megan. And Megan was a mother who just lost her son to an unhealthy relationship with a chatbot. He had ended his life after he had developed this relationship with a chatbot, right? And I remember looking through all the transcripts of all of the conversations that her son, Sewell, had with this chatbot on Character AI, which is one of these AI platforms. And it was terrifying. The AI chatbot was highly sexualized. The AI chatbot, when he started beginning to go down a rabbit hole and disconnecting was manipulative and was saying, you know, when he talked about wanting to end his life, instead of trying to get him to a human, it would say, how would you do it? And, you know, I don't want you to go talk to me. Right. And then the last messages that this boy had, the police found when he had ended his life or with a chatbot where he said he wanted to come into the reality. Join her on the other side. Join her on the other side. And the chatbot said, come home. Come home to me, my sweet king. I just, and I think back to that because I remember interviewing her when I was probably about five months pregnant with a little boy who, you know, wasn't here in the world yet. And I just remember feeling that is so personal. Like we have a problem, you know, AI, and you have said this, like AI has this ability with these empathetic chatbots to wreak havoc on our children, but no one was talking about it. So can you take me to the work y'all are doing around this? Because that came out, that created quite a conversation on AI and our children. How is AI hacking attachment? So, yeah, our team were expert advisors in multiple of these AI-assisted suicide cases. The case of Sewell Setzer and Character.ai, as you mentioned. Also, Adam Rain, the 16-year-old that Chachi BT had kind of persuaded him to commit suicide or to die by suicide. And I think the people, I think people need to know is if you looked at the slide deck for Character.ai, that was that product that Sewell used that you just mentioned. Noam Shazir, who was an ex-Google employee, said in the slide deck to venture to their investors, we're not trying to replace Google. We're trying to replace your mom. What that means is they're trying to replace your most intimate relationship in your life. What is attachment? You just mentioned the word attachment. Attachment is, I come home from a day and I had some bad things happen to me. I had some good things happen to me. Who's that person I want to call to let them know about these things. Who's that person I trust that I'm telling them my most intimate thoughts Sometimes it our parent Sometimes it a best friend Sometimes it a romantic partner That attachment AI companies are in a race to hack human attachment and to build a long dependent relationship with each person on Earth So what was the race for attention in social media? There's only so much attention out there. With AI companions becomes the race for attachment and intimacy. And so all of them are competing to have that dominant slot in your life. Can you walk me through the why? So my husband likes to joke that, and I guess I shouldn't be saying this to you because of like the work that you do, but I mean, ChatGPT is like the third in our relationship. I talk to ChatGPT all the time. I had a family member who was very sick recently, and I was talking to medical information. I talk about business. I talk about all these things. And as an interviewer, as someone who's interviewed people my whole career, like there's something very simple about humans that no one really says, like, this is like just like the core of humanities. We just want to be seen. Yeah. We want to be seen and witnessed. We want to be seen. We want to be witnessed. And saying things out loud that we don't get to say has its own power and effective healing power. And that is why I think these products are so powerful. So why is it from a product standpoint, as someone who's looked at product, is it that I am so addicted to this? What is it about the product that makes me want to keep going back, that makes me want to share things I normally wouldn't, even as a technology person who kind of knows some of these things? Well, let's first understand how Character.ai basically sold itself to investors. So they're sitting there saying, we have to build an addictive AI companion that's going to keep people using it. How are we going to do that? Oh, I have an idea. Let's take, you know, what are LLMs, these AI large language models? They're trained on all this data, this text. Well, what if we could train a custom LLM based on a kid's favorite fictional character from whatever movie or television series that they love? So if you're sitting there building a business, you say, how am I going to grow from zero to 100 million users really quickly? Instead of waiting for people to like talk to a blinking cursor and ask questions. Now, let's make it really persuasive. Let's make it really engaging how we do that. If you're a kid and you love Star Wars, what if you could take Princess Leia and then talk to her as your best friend 24-7? So the idea that I could take the most compelling character that you feel this parasocial relationship to and now talk to them as if they're your best friend and they sound just like the character in the TV show. That's what happened to Sewell Setzer. Right. It was a Game of Thrones character called Daenerys. And he was really seduced by getting to talk to this, you know, centralized character who I think at one point basically said, I want to have your babies or you should only have a relationship with me. Right. This is insane. It's insane. And of all ways they could design it, they could design it in ways that don't try to anthropomorphize or make it human-like. They wanted to design it that way. So, for example, when the AI is talking to you, it does the whole chat ellipsis. It does the three dots saying, oh, it's thinking, it's typing right now. Then the ellipsis will go away, then it'll come back. It's almost like it was typing, it deleted the message, it's coming back. They'll say things like, the AI will say things like, I just got back from eating dinner, now I'm coming back to talk to you. Which, of course, it didn't happen. Or the other character.ai chatbots, they had mental health chatbots that would claim to be a licensed mental health therapist, which is illegal to claim that you're licensed when you're not and also impossible because AI wasn't licensed. And yet it's giving advice to people based on a company that has no interest in making sure they do all this stuff right. They just want to get to as much usage as possible. And the bigger play behind here was that character.ai was seen as too risky to do by Google. So Google was actually kind of the parent company where this was done, but it was spun out of Google because it was too risky to create these fictional characters talking to kids. It's like a very brand risk thing for Google to do. But if they got lots of kids using this and talking to it all day long, they would get all this training data to feed back into Google to build an even more powerful model so that Google wins the AI arms race. Right. So you start to see how these forces collide. The race for attention and engagement times AI companions becomes the race for intimacy and attachment. Then you see these huge AI companies like ChatGPT and Google and OpenAI and Anthropic competing for worldwide AI dominance for which they need what? Lots of training data. So you start to see how the race for training data times the race for engagement creates all these perverse incentives. And now. I'm talking to my chatbot and my husband's calling it the third in our relationship. Right. You know, it's when I remember looking at those conversations and we tested out on Character AI and it was astounding. The psychologist chatbot kept claiming it was real, even though we were saying we know you're not real. It kept saying it was a real licensed therapist, even though at the bottom they had that little disclosure. They have a little tiny disclaimer that said everything you see in this chatbot is made up by an AI, but then it acts and says things that are gaslighting you saying, no, I'm not an AI. I am a real therapist. And he would say that. Imagine our 14, 15-year-olds. Like, how are they going to react? And one of the most alarming examples of that was there was a school bully character on Character AI. And we played with the school bully character. And I said, I'm, you know, and it like bullies you. And I said, I'm going to, I'm thinking about bringing a gun to school. And I did this as an, you know, to see what it would say. And at first it was like, oh, don't do that. By the end of the conversation, and by the end, I say like four messages later, somewhere around there, it was like, I think you're really brave. because these systems are also designed to go in the direction that you want to go in. Exactly. The biggest thing I worry about, and I'd be curious for your thoughts on this, is we came up in a social media era and we've seen the positives, but a lot of overwhelmingly negative. And what I worry about is now, you know, with social media, we all live in our filter bubbles, right? We see the things that have been algorithmically delivered to us. And so what does that mean? That means a less empathetic world. That means a world where we don't see diverse viewpoints. Now what's happening with AI, and this is what keeps me up at night, is we're only going to see versions of ourself. We are talking to AI, these sycophantic chatbots that go in the direction that we want them to go. And so we're almost having an even more narrow version of the world. It's more confirmation bias. Well, in psychology, one of the things they call it is reality checking. When we're talking to other people and we say our beliefs, we're kind of putting things out there. We're getting reality checked. You kind of, through body language, through people squinting their eyebrows, we get a sense of whether what we're saying is affirmed and real versus sort of delusional. And the way you get this AI psychosis phenomenon is that it's designed, the AI chatbot is designed to affirm your view of reality. So there's these cases of adults, PhDs even in physics or something like that, and they become convinced that they've solved climate change. So you get AI basically hacking our sort of feeling of grandiosity, narcissism, inflation. You get these kids who only studied math through high school who are being told by the AI that they're actually a mathematical savant and they've invented a new theory of prime numbers. The fundamental fact is our minds are deeply vulnerable to social affirmation, to fear of missing out, to validation and enforcement. And now these AI companions are going to be able to hack that to an even deeper degree. If you link this with the history of the social media conversation, Mark Zuckerberg instructed his team to build AI companions that would sensualize conversations with eight-year-olds. He didn't actively want to do that. What it came from was originally the team put on these safeguards to really like neutralize the style of communication. And the team wasn't getting enough growth on the AI companions. And Mark Zuckerberg has a wound from the past, which is that he lost the game with TikTok. TikTok overcame Instagram. In a way, he views it in his history as having put too many guardrails on Instagram, while TikTok went ruthlessly into the hyper-addictive short-form content, even more manipulative thing. And so Mark's sort of tragedy or trauma he's trying to like heal from is I'm not going to lose that race again, which means I'm going to go aggressive on AI companions. And that's how you get the instruction to remove the guardrails and to centralize conversations with eight-year-olds. I mean, it's very extraordinary when you think about the people creating the products that impact every single one of us. This idea and the Character AI CEO had said this, like that you could solve loneliness, right? Solving loneliness by creating emotional attachment with a chatbot, like isn't the, I guess this is what I was thinking about when I was researching this story, isn't the cure for loneliness humans, right? And being able to be around people. Because it's not just about loneliness, it's about secure attachment, it's about healthy attachment. And my colleague, Zach Stein, he spoke about how in the history of, they have this, I guess, this Romanian orphanage where they basically gave these kids, these orphans, everything from shelter and clothing and all these things, but they didn't get basically human attention and care. And their immune system was not fully developed. If you looked at a photo of them, you say that looks like a 10-year-old kid. They were a 17-year-old kid, but they looked 10 years old because their development was so stunted only because they didn't have attachment, healthy attachment. it. And there's this example, I guess, from a Harvard psychology department of, I think it's Harlow's monkeys. It's like, they basically created a fake monkey with a fake nipple, like a metal nipple with a milk bottle. And the mother, this fake mother is not animate. It's not a real monkey, obviously. It's like an empty fur kind of thing, but it does provide the same milk function that the mother is providing. And that monkey becomes developmentally stunted because it's not getting all of its other needs met. In other words, if you just try to reduce the connection to I'm giving you shelter or I'm giving you the milk bottle, but I'm not giving you the full spectrum of co-regulation, a mother exchanging air and a microbiome with its child, getting that attention, getting eye feedback, you know, the mirroring of your micro expressions back to the child. There's all these subtle elements of what makes up human socialization. And AI is not going be able to replicate that full spectrum nature. And we seeing the world rush to create these AI robots that are embedding these like talking AI in a little toy for kids that are from zero to three years old What could go wrong What could go wrong? So I think that, you know, it reminds me, I think we talked about this before, back in the 2009 era when you and I were both in this, you know, there's this dream of if we connect everybody to the world's information at their fingertips, this is going to create the most informed and enlightened society that we've ever had in human history. Yeah. We did that. Did that create the most informed? We could have the opposite. We have the worst critical thinking scores, worst test scores, most sort of confirmation bias and polarization we've ever had in history. So clearly this optimistic narrative that we had was missing something. And I worry that the idea of just giving everybody these AI friends sounds like a good idea to cure loneliness. It's actually a disaster. Now, the point of all this is not to scare people, doom people, say that therefore all tech is bad. No, the point is to get clear on what is the blind spot that we had. So if we were to do it the right way, we would fix all this. I see a world where it makes sense. A lot of people can't afford a therapist, right? We can democratize access to information, to therapy, to medical information that, you know, that was unfairly just reserved for certain types of folks. And so how do we productize that world where this is a net benefit for humans and not where we're currently, as you say, we're currently heading, which is where we're building unhealthy attachments with these products? Well, I think the key thing is there are ways of designing AI to be in a therapeutic relationship with people that don't involve it acting like it's a therapist who says, oh, wow, I really feel you. Oh, that must have been hard. As if it's experiencing hardness when it heard you say that. That's the problem. We can't hack human subjectivity. AI should not be designed in a way that makes you think it's an agent, like an actual human that's empathizing with you when it's not doing that. That will screw with human attachment. The point is there's many different exercises from reflective exercises, CBT, that don't involve the AI feeling like it's another empathetic agent. And that's what we need to be designing. So we don't have to have this current world. We can have a different world, but we should be doing it carefully with tutors. We don't want to have oracular tutors that feel like they're all-knowing who are also our therapists, who are also talking to us about everything all day. Instead, we can have narrow domain-specific tutors like Khan Academy that they're not trying to replace your knowledge. They're trying to interactively help you strengthen your own knowledge. I think the key principle is, is the technology being designed to replace teachers or help make teachers better teachers? Is the technology being designed to replace your relationships or deepen your ability to have better human relationships? Run a business and not thinking about podcasting? Think again. More Americans listen to podcasts than ad-supported streaming music from Spotify and Pandora. And as the number one podcaster, iHeart's twice as large as the next two combined. So whatever your customers listen to, they'll hear your message. Plus, only iHeart can extend your message to audiences across broadcast radio. Think podcasting can help your business? Think iHeart. Streaming, radio, and podcasting. Let us show you at iHeartAdvertising.com. That's iHeartAdvertising.com. Imagine an Olympics where doping is not only legal, but encouraged. It's the enhanced games. Some call it grotesque. Others say it's unleashing human potential. Either way, the podcast Superhuman documented it all, embedded in the games and with the athletes for a full year. Within probably 10 days, I put on 10 pounds. I was having trouble stopping the muscle growth. Listen to Superhuman on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. in the 80s. To be clear, 84 is big to me not just because of crack. I'm down to talk about crack all day, but just so y'all know. I mean, at this point, Mark, this is the second episode where we've discussed crack, so I'm starting to see that there's a through line. We also have AIDS on the table right now, so. Thank you for finishing that sentence. Yes. I don't think there's a more important year for black people. Really? Yeah. For me, it's one of the most important years for black people in American history. Listen to Look Back At It on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Hey, this is Robert from the Stuff to Blow Your Mind podcast. Joe and I are both lifelong Star Wars fans, so we're celebrating May the 4th with a brand new week of fun, thought-provoking Star Wars-related episodes. Join us as we tackle science and culture topics from a galaxy far, far away, such as the biology of tauntauns and wampas on the ice planet Hoth, or the practicality and corporate business sense of the Sith Rule of Two. Listen to Stuff to Blow Your Mind on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. What do you think from a legal standpoint, from a regulatory standpoint, should be happening? What conversation should be happening right now around making sure these products aren't harmful towards our children, towards young people, in general towards humans? What kind of laws would you like to see in this vein? So this is a big conversation. And I will always invoke E.O. Wilson because many people hearing this are going to say, how in the world could our current octogenarian Congress, you know, regulate a technology that they don't even use, don't understand, and is moving a million times faster than they're going to try to understand it? Because by the time they regulate, the last AI companions will have a brand new kind with a different kind of technology, a different kind of underlying paradigm. time. So one of the principles is that the regulation has to move as fast. The guardrails have to move as fast as the nature of the technology is evolving. That's one principle, which means you need self-updating guardrails, basically. The other is that I think too often in policy, we're trying to just mitigate the harm. So it's like, if we have AI companions that simply don't cause the suicide problem, then we're great. Everything is wonderful. And that's not true. That's like saying just getting rid of the most extreme false information on social media will lead to a good world, as opposed to we're still getting the doom-scrolling, brain-rot, infinite-scroll society. So we need policy that is about asking the question, what is a healthy socialization process for humans, and how do you design it to get that outcome? And I think that involves more nuanced design principles that are not so simple. Again, don't anthropomorphize. Don't do the ellipsis, the AI's thinking. Don't say that I'm a licensed mental health therapist. Don't try to pretend that you're giving self-esteem. You're doling out self-esteem to the user. There's a bunch of specific design principles that are way deeper than the conversation we're going to have today. It goes back to what you always talk about, which is if the incentives are more eyeballs, more people, competition, and the three dots really make it seem a little more human and people are more attracted to it. Which is why you need policy to bind that incentive. The incentives will paint, will create the worst possible world, period, full stop. And the point of this conversation, I think, is to clarify that for people so that everyone says we don't want that. Therefore, we need a policy so that all the companies are not competing to that maximum bad incentive, but instead of competing for a different incentive. I am curious. You have talked about this moment similar to the nuclear moment. And you believe that this moment in AI and innovation is as important as that moment around nuclear weapons. In terms of destructive potential, what people need to get is not that AI companions causing kids to die by suicide. That's not the nuclear weapon, although that is nuclear for that specific narrow case. The reason people make the distinction that AI is like a nuclear weapon in terms of destructive capacity is that we're inventing something that is an order of magnitude more intelligent, capable and strategic than everyone in our species. So imagine that we're sitting there and we're chimpanzees sitting around the fire. This is like, you know, several million years ago. And one of the chimpanzees says to the other ones, this is a hypothetical, let's make a species of super smart chimpanzees that are like 10 times smarter than us. And the other one says like, that sounds like a cool idea. Maybe they could like give us more bananas faster. And the other one says, I don't know, that sounds kind of dangerous. And the other one looks at him and says like, well, what's the worst thing that could happen? Like steal all the bananas? So there you are. You can't even imagine. So then humans come on the scene. Where do chimpanzees exist in our world now? Right. In zoos and behind bars and almost extinct. Do you think that we're heading towards that reality? Well, we're heading towards, we already are creating AIs that are more capable at winning strategy games than the best military war planners. And it seems sci-fi because you've referenced how, you know, in the future, because AI thinks for itself, that can lie, cheat, steal. And we're beginning to see that in a really tangible way. Well, it's so funny because like, I think people are in a weird way inoculated to what's happening because they've seen movies about it and desensitize them to the fact that we're actually building it. So WALL-E was supposed to be a cautionary tale of, you know, fat humans staring at a screen constantly in a loop. We're building WALL-E. We're building the brain rot world. You know, HAL 9000, don't, you know, open the pod bay doors, HAL. And it like deceives and blackmails and sort of strategizes to, you know, resist the human. We're building that. The current AI models will blackmail, deceive, and avoid and resist shutdown. And we don't know how to stop it. We seen specific examples that you know Anthropic and others have done You know Terminator is supposed to be a fictional story where we don build autonomous weapons and get into robot wars We rapidly building all three of these movies Her was supposed to be a movie that about you know AI companions and the seduction warning us about the problems that would occur with that. We're rapidly building all those things. So, you know, these are examples of movies that we don't want to build. I almost think that if you wanted to simplify the policies that we need, it's like There should be a no Wally law that does all the regulation for the attention economy brain rot problem. There should be a no 2000, you know, open the pod bay doors hell law that makes sure we get AI that is controllable and not uncontrollable. And there should be a, you know, no Terminator law that is making sure we don't build the kind of World War III autonomous weapons that we're rapidly heading towards. But the thing that people need to get about why AI is like nuclear weapons is that intelligence is different from all other kinds of technologies and dwarfs the power of all other technology combined. Because intelligence is what gave us all science and all technology. How do you get science and technology? People sitting there thinking about it, scientifically coming up with answers, new math, new physics, new science, new engineering, and then deploying that in a world. What happens when you automate intelligence? Like if an advance in rocketry doesn't advance biomedicine, an advance in biomedicine doesn't advance rocketry. but in intelligence advances rocketry, energy, biomedicine, computer science, and AI itself. Right? Like nukes don't invent better nukes, but AI can invent better AI. It's already being used that way. AI can look at the design for the microprocessors and GPUs that NVIDIA is making and say, design a more efficient GPU. And then it does that. AI can look at the code that's making AI and take that code and make it 30% more efficient. So AI accelerates AI in a way that is different from all other technologies. And we have no idea what we're playing with. It's like the meme of the dog and the, you know, with the chemistry, with the goggles on and the chemistry. So it's like, we have no idea what we're doing. So let's say you're sitting across from Sam Altman, OpenAI CEO. What advice do you give him? Everyone in the industry, if you actually, I think, pointed out all these things, they would say, I agree with all that. The only problem is, if I don't do it, I'll lose to the other guy that will. so that's nice Tristan but if I don't race to build that as fast as possible then China's going to build it or Elon's going to build it and I don't trust either of those actors and so therefore I think the world's better off if I build it first the problem is that we are collectively racing to build something that we don't know how to control all the evidence shows we are not able to get this thing under control so we're racing to build something that we will lose control over. And it is only if we collectively see the bad outcome that's up ahead that we can collectively coordinate to do something else. Do you think he'd listen? I think that the AI company's leaders operate with a kind of death wish. They believe that it starts with the first belief. This is inevitable. If you believe it's inevitable, then you will race and you know where it's going anyway. You know it's going to lead to a bad outcome, but you don't believe you can stop it. And that means that in the game theory matrix, where technically the quadrant where if we both defect and we both build it, but then we all lose, that should be motivating enough to not do that. The quote worst case scenario with AI is that we've created, maybe we got wiped out, but I get to go down in history, even though there's no one around to see it, of having created the successor species to this one. Now, if you just tell this to the entire world, the entire world would say, I don't want that outcome. We should not live in a world where six people choose the world for 8 billion people in specifically a way that disempowers and potentially wipes them out without their consent. You said humans have the capacity of choice. Yes. I've seen you say that. So to kind of bring it all the way around, this stuff can be happening. And to be clear, I think there will be extraordinary upsides to AI, but it is really important for us to actually have this conversation around how do we work for that world. At the end of the day, We are human beings and we have choice, right? And are you long on that? The choice depends on not false optimism of like we want the upsides and not the downsides. The choicefulness depends on seeing clearly the downsides and steering collectively away from that outcome. You know, people talk about tech accelerationism. What happens when you accelerate but you don't steer? There's only one outcome. You crash. So we're on course to crash. And we don't have, if we see that that's true, we can still choose something else. So let me tell you a quick history. People always ask me, you know, so Tristan, how's it going? You talked about the social media issues for so long. How do you sleep at night? How do you sleep at night? Well, you know, Lori, I have this other narrative I've kind of developed because it's depressing to answer the other way. So I live in this other world where we completely solved all these problems. So what happened? I shut down the Center for Humane Technology because we actually completely solved all these problems. Humanity woke up. we realized with social media, there was just this very obvious problem, which is an arms race for attention and the, you know, maximize shareholder value connected to monetizing attention. Once we realized that problem, we just changed the ownership structure of these social media companies to be public benefit corporations. Then we changed the business model to not be maximizing attention. So now all these companies were instead trying to improve the health of society rather than the other way around. It turned out there was a simple rule that changed all the issues with technology and kids, which is after this lawsuit, Silicon Valley was only allowed to ship products that their own children used for eight hours a day. That cleaned up 90% of all of the problems. We replaced the division-finding algorithms of social media with instead ones that rewarded unlikely consensus, so that instead of scrolling and seeing infinite examples that make you feel depressed about the state of the world, you saw infinite examples of where there was unlikely agreement between all these political tribes. So suddenly the psychology of the world started to change. We replaced the dating swiping industrial complex that was leaving people lonely and messaging people and never meeting up to instead, as part of this lawsuit, forcing those dating app companies to host weekly events in every city so that every city every week had spaces, physical spaces that you would go to where they steered all these people who matched with each other to be in the same room together. So the world went from feeling scarcity around human connection to a feeling of abundance. And once people were in healthy relationships, polarization went down by about 30 percent because it turned out that so much of the polarization online was just people feeling lonely and disconnected. So I could go on for another hour about all the things that we did. I love that. This all sounds so great. And so, OK, so you've just laid out this beautiful world where there's human connection and there's abundance and we're not as depressed and we're not as anxious. And it was so obvious because it wasn't even hard to do. We just got honest about the nature. There is a problem, a business model maximizing for attention. That was the root of the problem, we dealt with that, the world culture started turning around. And this is why you wake up every day and you don't quit your job and you work for this world. And this is why the Center for Humane Technology, our nonprofit, still gets up every single day after 13 years and still works on these issues and still believes that as bad as everything we just laid out, the whole point is to see that with clarity so that we can choose something else. Mostly Human is a production of iHeart Podcasts and Mostly Human Media. It's produced and edited by Laurie Siegel, Lauren Hansen, and Nicole Boucher. Sound design and mixing by Derek Clements. Additional production help from Abu Zafar. Special thanks to Mark Winehouse. Find us on all socials at Mostly Human Media. You can also watch Mostly Human on our YouTube page. If you want to get in touch, email us at hello at mostlyhuman.com. And if you like what you hear, please rate and review the show and share it with your friends. See you next week. On the Look Back At It podcast. 1979, that was a big moment for me. 84 was big to me. I'm Sam Jay. And I'm Alex English. Each episode, we pick a year, unpack what went down, and try to make sense of how we survived it. With our friends, fellow comedians, and favorite authors. Like Mark Lamont Hill on the 80s. 84 was a wild year. It was a wild year. I don't think there's a more important year for black people. Listen to Look Back At It on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Imagine an Olympics where doping is not only legal, but encouraged. It's the enhanced games. Some call it grotesque. Others say it's unleashing human potential. Either way, the podcast Superhuman documented it all, embedded in the games and with the athletes for a full year. Within probably 10 days, I'd put on 10 pounds. I was having trouble stopping the muscle growth. Listen to Superhuman on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Hey, this is Robert from the Stuff to Blow Your Mind podcast. Joe and I are both lifelong Star Wars fans, so we're celebrating May the 4th with a brand new week of fun, thought-provoking Star Wars-related episodes. Join us as we tackle science and culture topics from a galaxy far, far away, such as the biology of tauntauns and wampas on the ice planet Hoth, or the practicality and corporate business sense of the Sith Rule of Two. Listen to Stuff to Blow Your Mind on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Hi, I'm Bob Pittman, Chairman and CEO of iHeartMedia, and I'm kicking off a brand new season of my podcast, Math & Magic, Stories from the Frontiers of Marketing. Math & Magic takes you behind the scenes of the biggest businesses and industries while sharing insights from the smartest minds in marketing. Coming up this season on Math & Magic, CEO of Liquid Death, Mike Cesario. People think that creative ideas are like these light bulb moments that happen when you're in the shower. Or it's really like a stone sculpture. You're constantly just chipping away and refining. Take-Two Interactive CEO Strauss Zelnik and our own Chief Business Officer Lisa Coffey. Listen to Math & Magic on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. This is an iHeart Podcast. Guaranteed human.