Feed Drop: "Into the Machine" with Tobias Rose-Stockwell

Hey everyone, it's Tristan Harris. Welcome to Your Undivided Attention. And today we're going to bring you something a little different. This is actually a conversation that I had with my friend Tobias Rose Stockwell on his podcast called Into the Machine. And Tobias is also the author of the book The Outrage Machine. He's been a friend for a long time. And so I thought this conversation was honestly just a bit more honest and sobering, but also hopeful about what choice we could really make for the other path that we all know is possible with AI. So you may have noticed on this podcast, we have been trying to focus a lot more on solutions. We actually just shipped an episode last week, which was what if we had fixed social media and what are all the things that we would have done in order to make that possible. And just to say, we'd really love to hear from you about, you know, these solutions and what you think the gaps are and what other questions you're holding. One of the things about this medium is we don't get to hear from our listeners directly. And we'd love to hear from you. So please, if you have more thoughts, more questions, send us an email at undivided at humanetech.com. And I hope you enjoy this conversation with Tobias. There's something really strange happening with the economy right now. Since November of 2022, the stock market, which historically linked up directly with the labor market, diverged. The stock market is going up, while job openings are going down. This is the first time this has happened in modern history. Office construction is plummeting. Data center construction is booming. And if you look closely at where the money is moving, in the world of investing, a lot of people are betting on the fact that AI workers will replace human workers imminently. My guest today is Tristan Harris. He's the founder of the Center for Humane Technology. You may have seen him in the Netflix documentary, The Social Dilemma, which he produced and starred in. Tristan has been a champion for AI ethics for a long time. And this conversation gets strange. Tristan and I don't agree on everything, but I think we land somewhere important, which is discussing pragmatic solutions that might be possible in this very strange moment with AI. I really enjoyed it. I hope you will too. A few notes. We speak about different AI companies. My wife works at Anthropic. The CEO of Anthropic is named Dario Amadai. So with that, I'm Tobias Rostockwell. This is Tristan Harris, and this is Into the Machine. Tristan Harris. Good to be with you, Tobias. Thanks for being here, man. Always. We've been talking about these issues for a long time. I'm really a big fan of you and your work and the book The Outrage Machine and the public advocacy you've done to help people understand these issues. Same. Absolutely. You've been such a force of nature making these issues visible to the wider public. So you've done a great job of injecting yourself into the current discourse recently talking about AI. Where do you land in terms of AI takeoff right now? Where do you see things in the next three years, five years, ten years? I think I don't spend my time speculating about exactly when different things are going to happen. I just look at the incentives that are driving everything to happen and then extrapolate from there. So you don't have to go into the futures of takeoff or intelligence explosion. You can just look at today. We have, as of last week, Cloud 4.5 can do 30 hours of uninterrupted complex programming tasks. That's just like letting your AI rip and just start rewriting your code base for 30 hours. Today, Claude is writing 70 to 90% of the code at Anthropic. So when people talk about takeoff or just some kind of acceleration in AI progress, well, if you have AI companies, the code that's being written is 70 to 90% by the AI. That's a big deal. Today, we have AIs that are aware of how to build complex biological weapons and getting past screening methods. Today, we have AI companions that are driving kids to commit suicide because they're designed for engagement and sycophancy. Today, we have AIs that are driving psychosis in certain people, including an investor of open AI. So these are all things that are happening today. And today, as of actually just two weeks ago, we have these AI slop apps that are trained on all of those creators. And they're claiming to build AI and race to superintelligence so they can cure cancer and solve climate change. But clearly, I think the mask is off. They're releasing something just to get market dominance. and the more shortcuts you take, the better you do at getting to that goal. And so I really do think especially that these AI slop apps, like a lot of people, if you look at the top comments when people see these videos, it's like, we didn't ask for this. Why are we getting this? And it's so obvious that the thing that we've been saying for a decade, which is if you show me the incentive, I will show you the outcome. If there's an incentive for market dominance and getting users and getting training data and using that to train your next AI model, you're going to take as many shortcuts to get there as possible. So I'm going to push back on you a little bit here. Yeah. We've been friends for a long time. One of the first people we talked about the attention economy. We've been talking about this stuff for over a decade now. So going back to the kind of early conversations about this and the discourse that we were a part of in those early days, one of the things that you zeroed in on back then was advertising-based business models. This is clearly not the case with current LLMs. In fact, Sam Altman follows you on Twitter. If he was to have tracked Tristan's talking points over the last 10 years, you would think in the design of ChatGPT, he would have been orienting around some of those lessons. It's a subscription-based business model. It's trying to be as useful as possible. if the business model is the primary incentive for a product's development, what are they doing wrong with LLMs? And what is the right business model? So this is partially right. So Sam Altman, I think, himself actually said, basically recapitulating what we said in 2014, which is that social media was the first runaway AI optimizing for a narrow goal of engagement and time on site and frequency of use. And that was sort of a narrow, misaligned AI that wrecked society because it optimized for addiction, loneliness, personalized inflammatory content that divided society, personalized for every single political tribe that's out there. I think that he actually agrees with, and I happen to know, did very much agree with that diagnosis as early as 2016, 2017. But I think it's important to zoom out when you look at ChatGPT that it's not just the business model of a product. It's their overall goals. What is their actual incentive? And it's not just their business model. Their actual incentive is to get to artificial general intelligence. I'm saying that because that's literally OpenAI's mission statement. So how do you get to artificial general intelligence? Well, you need market dominance. You need as many people using your product for as long as possible because you use that to get as much usage, to get as much subscription revenue, to prove to investors. You use the fact that you're at the leading edge of the race to attract the best engineers and the best AI talent because they want to work at the leading AI company, not the third best AI company. And you use the investor dollars that you raise from all of that activity to fund the creation of new GPUs and new data centers and use the GPUs and the training data to train again the next model and you rinse and repeat that flywheel. so that's their real goal and they will do everything in their power to maximize that usage and engagement so it is true that they pride themselves in saying look we're not like social media we're a tool we just want you to use it but you'll notice i think it was the atlantic just a few weeks ago there's a writer who coined the phrase not clickbait but chat bait if you're using chat gpt and you ask it a question and then it says well would you like me to put all that information into a table for you and then turn it into a diagram or, you know, and you're like, well, actually, I really would like you to do that. And the reason that they're doing this chat bait, not click bait, is they're baiting you into more engagement and more usage. Now, you would say that that actually is helpful because the thing that they're baiting you with is something that would actually further assist you on the original task that you're on. But they're still just doing that to basically show that they have lots of usage, build a habit, have you feel like you need to deepen your reliance and dependency on this AI. And that still does generate incentives for sycophancy or flattery. So the AI is much more likely to say, great question. I totally agree with you. Let's go into that versus saying, actually, there's some problems with your question. Let me be a little bit disagreeable. The disagreeable AI doesn't compete as well as the agreeable AI. And so we're already seeing the effect of that agreeableness turn into these AI psychosis. That's the broad term for the phenomenon, but basically people who are having a break with reality because the AI is just affirming their existing views. Including one of the open AI investors, I think it was Jeff Lewis, started going crazy because he'd been talking to it. So it shows you can get high on your own supply. So is there a better business model for these tools? Well, I think it's good relative to worlds we could be living in. It's good that we are living in the world of subscription-based revenue for AI products. But it's also important to note, I believe OpenAI hired, I forgot her name, Fiji something, who used to be the head of product at Facebook. And I think you're already starting to see her influence at the company, including the fact that OpenAI did not have to launch an AI slop TikTok competitor that has short-form AI-generated videos. But I think that is an example of that influence. And also when you have a leader at the company who's making product leadership decisions, who spent the last 15 years working at a company that was entirely built around engagement, it's like paradigmatically the sense making and choice making that you are doing is subtly infused with the logic of I need to get people's attention. And so I think we are starting to see those kinds of choices. We don't have to go down that path. We shouldn't go down that path. But engagement in advertising is only one of the many issues that we have to deal with with this race. I'm thinking through how it might be done differently. We have trillions of dollars of investment in this new technology. We want it to be maximally beneficial to humanity. I certainly understand the longer-term goal of trying to mitigate fast takeoff scenarios in which we're left with loss of jobs and all these other things. I'm curious what form this tech would take if it was designed to maximally benefit humanity, in your opinion. We were talking about earlier that it's not about the business model for how you pay for your OpenAI chat subscription. That's just to get some revenue along the way. If you're OpenAI or Anthropic, and you've raised hundreds of billions, if not going towards trillions of dollars, to build out these data centers, how are you going to pay that back? The answer is you have to actually own the world economy, meaning own all labor that is done in the economy. Just make it very simple for people. Imagine some company, Acme Corp, and it has 100 employees. Right now, it has to pay those dollars funneled down to the 100 different employees. AI, country of geniuses, shows up on the world stage and it says, hey, Acme Corp, you could pay those employees $150,000, $100,000 a year, grow humans over 20 something years, have them go to college. They might complain. They might whistleblow. You have to pay for healthcare. As an alternative, you could pay this country of geniuses in a data center for less than minimum wage. We'll work at superhuman speed. We'll never complain. We'll never whistleblow. You don't have to pay for healthcare. And they'll do the same work, especially as the entry-level cognitive work of your company. Super cheap. What is your incentive as a company? Is it to protect all your employees or is it increased profits and cut costs? So you're going to let go of all the junior employees and you're going to hire these AIs. As you hire the AIs, that means the money that used to go to people is starting to progressively go towards this country of geniuses in a data center. So we are currently heading, if you just look at the obvious incentives at play, for all the money in the world to instead of going to people, will get increasingly moving towards these AI companies. When Elon Musk says that the optimist robot alone will be a $25 trillion market cap product, what he's saying is the labor economy is something like $50 trillion. He's saying we're going to own the world physical labor economy. I would ask when in history has a small group of people ever concentrated all the wealth and then redistributed it to everybody else? doesn't happen very often. So again, I'm not making predictions about AGI or takeoff or superintelligence. I'm literally just looking at how does the system evolve? And of course, the AI companies will never talk about it the way I just talked about it. They'll talk about it as we're going to automate all this work. We're going to get this huge boost in GDP growth, which historically, if GDP went up, it's because also all of us were doing better because we're getting the rewards of that. But suddenly we're talking about a new world where GDP is going up way more, but it's not coming to real people. Because it's going to these handful of companies, the geniuses in the data center. I'm thinking about some of the studies that have been done on ChatGPT and worker productivity, in that it tends to be very helpful for people that are junior workers and that don't necessarily have high levels of skill in a topic. And it actually brings them up to an average baseline across pretty much any task they're trying to do. It's dramatically helpful for them. But for more senior employees and more expert level producers in the economy, it actually brings them down and it actually causes them to spend more time editing the tools, working with them, trying to figure out how to work them into their existing workflows. So in some ways, this is actually quite an egalitarian technology if you look at how people are using it presently, right? I'm familiar with some of the product teams at particularly Anthropic right now who are really trying to do the best they can to make sure this is aligned with human flourishing. I'm curious what you would potentially say to them because they're asking these questions on a daily basis. They're very familiar with your work. They want to make this stuff maximally beneficial. I totally believe that, by the way, especially with Anthropic's case. I it's easy to this is not a critique of evil villains running companies who want to wreck the world. It's just we all have to be as clear eyed as possible about what incentives are at play. And Anthropic, I think, has done almost the best job of warning about where these incentives take us. I mean, I think Dario basically said we're going to wipe out like 50 percent of all entry level work in a very short number of years. He's one of the few executives that's actually willing is willing to say that. Exactly. Exactly. The previous situation is you have these AI company CEOs who behind closed doors know this is going to wreck the economy and they don't know what's going to happen. They don't have a plan, but they're not evil for doing that. Their logic is, so it starts with the belief, this is inevitable. If I don't do it, someone else will build AI first and will automate the economy and steal all those resources and maybe it'll be China, so therefore the US has to do it. Third, I actually believe the other people who might build AI, if I don't, have worse values than me. And so actually I think it be better if I built it first Therefore I have a moral duty to race as fast as possible to build it before the other guys do No one likes the collective shortcuts that are being taken to get to that outcome But everyone because of these sort of fractal incentive pressures it forcing everybody to make choices that ironically make us all bad stewards of that power Like one of the reasons you don't want them to make it is you don't trust them to be a good steward of that power. But ironically, for me to beat them and get there first, I have to embody ways of being and practices that embody not being a good steward of that power myself. But for the fact that there was a race, I think everybody would agree that releasing the most powerful, inscrutable, uncontrollable technology that's already demonstrating behaviors like blackmailing engineers or avoiding shutdown and releasing this faster than release any other kind of technology we've ever had before, everyone would agree this is insane, but for the fact that there's this race pressure pushing us to do it this way. And I think that it's like a frog boiling in water. Like we're all just sort of living it and suddenly Chachapichich has got 10 times smarter and suddenly it's doing more things and suddenly jobs are getting displaced and suddenly kids are getting screwed up psychologically. It's just all happening so fast that I think we're not pausing and saying, is this leading to a good place? Are we happy with this dynamic? It's like, no, this is insane. You should never release a product, I mean, a technology this powerful and this transformative this quickly without knowing how you're going to care for the people on the other end. But again, it's really important to note that if the ultimate prize or rather the ultimate logic is if some worse actor gets this very transformative kind of AI, let's just call it transformative AI, that they can snap their fingers and build an army of 100 million cyber hackers like that. And then it is better at all humans and programming and hacking. And you can unleash that on another country. Well, that risk alone, just that one is enough to justify me racing as fast as possible to have those cyber capabilities to try to deter the other guys from having that. And so really, I think there's a lot of good people who are all caught in a race to this outcome that I think is not good and not safe for the collective. It's inviting us to a more mature relationship with the way we deploy technology in general. I respect the people at Anthropic enormously, and they have started by saying that the way that the other people building AI is unsafe, and that's why they started doing it their way. In fact, that was the original founding of OpenAI as well is we don't trust Larry Page and Google to do this in a safe way. He doesn't actually care about humans. That was the conversation that Elon had. So ironically, there's a joke in the AI safety community that the biggest accelerant of AI risk has been the AI safety movement, because it causes everyone to take these actions that lead to an unsafe outcome. And one of the things about Anthropic is that there are some who argue that the fact that they're so known for safety creates a false sense of security and safety because people just assume that therefore there's this one company, they're doing it safely, we're going to end up in this positive result. But they're the ones doing the research and leading on the publishing, showing that their current AI models are uncontrollable and will blackmail people when put in a situation where the AI is sort of being threatened to be replaced with a new model. Let's still imagine that for a second, because it seems like everyone on Anthropik's PR team would probably be against sharing that kind of information, for instance, right? There is some substantial courage it probably takes internally. Establish a baseline of saying, look, we're going to actually be as wide open as possible about negative capabilities here. I hope you didn't hear what I was saying differently. We should applaud the fact that they're taking those leading steps. I'm just naming one other secondary effect, which is if people, some people believe that them being known as safety, assume that therefore the actual implementation, we should just deploy that as fast as possible and it'll be okay. We can deploy that in our military systems. And it's like, just because they care more about safety doesn't mean that they've solved the problem and it is safe. So it does suggest, at least, part of the discourse is around the problematic capabilities of these tools. And Dario has this line about trying to make a race to the top. You talk about a race to the bottom of the brainstem. He's trying to... Race to the top for safety. Race to the top for safety. I think their assumption is, you're not going to stop this train of investment and research and capabilities improvement, that the only way to get ahead of it is to build a frontier model and then red team the hell out of it. Build as many deep tests for flaws and for negative capabilities as you can potentially extract from it and then publish those as widely as possible. Personally, that actually makes some sense to me, I would say, that just to kind of lay my cards on the table, there's this narrow path in the middle, which is we need to figure out how to make sure these tools are actually safe before we deploy them to the widest audience possible. I don't know how you do that without an actor like Anthropic potentially trying to test these things aggressively and taking this investment to build these very highly capable models. There is something about their most recent model, which I find interesting, is that it is safer. Their safety, I've used it a bunch, and it's actually frustrating much of the time. And there is this thing where you're working on it, and then you trigger a safety response, some kind of red line, and then it says, I'm sorry, I can't answer that question. Well, ChatGPT will answer that question for you. And so I immediately go to ChatGPT. is actually the most permissive when it comes to this stuff. So there's this natural dynamic there. And again, you can run these models, some of these models locally. They're getting smaller and smaller and more capable. They're getting more and more powerful. Once these models are actually out in the world, we're not going to be able to clamp down on usage of them. So as soon as there is a highly capable model that's out there, it's going to be available to people. And people are going to circumvent and try to avoid censorship. What's worse is that people will say, I'll make something as powerful as that one, but then I'm going to take off all the safety guardrails because that'll make it the most free speech AI. There's a model for that right now. There's a couple of companies that are actually promoting that as their primary market edge. But to be clear, we're not talking, often safety gets reframed as, does the model say a naughty thing or not? But actually building on the example you're giving of Anthropic, my understanding is the latest model, the good news is when you put it in that situation where it's going to get shut down and will it blackmail the employees, they have trained it now in a way where it does that less often than before. The bad news is that the model is now apparently way better at situation awareness of knowing when it's being tested and then altering its behavior when it thinks it's being tested. It's like, oh, you're asking me about chemical, biological, radiological risks. I'm probably being tested right now. I'm going to answer differently in that situation than I answer in other situations. The main thing that is just like crystal clear that people need to get is that we are making progress in making these models way more powerful at like an exponential rate. we are not making exponential progress in the controllability or alignability of these models. In fact, we demonstrably, because of the evidence that Anthropic has courageously published, we know that we still don't know how to prevent self-awareness or prevent deception or these kinds of things. It's great that they're working on it. And to steelman what you're saying, if we lived in a world where there was no Anthropic, then you'd have companies building all of this the same way, but maybe other companies would not have prioritized demonstrating scientifically that these risks are real. So they would publish them. Yeah, exactly. Because so given our alternatives, you had maybe Eliezer or Nate on this show who wrote the book, If Anyone Builds It, Everyone Dies. It's a very provocative and extreme title. Many ways that people try to say we need to do something differently with AI or go more safely is based on using arguments. They ask you to logically deduct and get to an outcome, a conclusion, that says that this is a dangerous outcome, therefore we need to stop or we need to pause or we need to coordinate or something. But we've all seen how unsuccessful arguing about this has been. From one perspective, you can say that anthropic is just a crazy multi-billion dollar alternative way of just simply demonstrating the actual evidence that would have us successfully be able to coordinate or slow down or figure this out. It's an interesting angle. I think that at the end of the day, all of this depends on, you know, will people keep going? It's like, if this was actually a nuclear bomb that was like blowing up in T minus 10 seconds, the world would say, no, let's prevent that from happening. But if a nuclear bomb was blowing up in 10 seconds, but the same nuclear bomb in 10 seconds because it's also going to give you cures to cancer and solve climate change and build unbelievable abundance and energy. What would you do with those two things hitting your brain at the same time? You and I have talked about how our brains process information for 10 years, and so much of the social media thing was that. Let's look at the object of what AI is. It's both a positive infinity of benefits you couldn't even imagine, of invention and scientific development that we literally cannot conceptualize, you or me, or even the most aggressive AI optimist cannot conceptualize what something smarter than us could create as a benefit. So I think the optimists are underselling how amazing it could be. But at the same time, AI represents a negative infinity of crazy things that could also go wrong. So I ask you, is there a precedent for something that is both a positive infinity and a negative infinity in one object? Do we have anything like that? The closest example is probably nuclear energy. That's not like the ability to generate everything. Because it's like, imagine we're a bunch of chimpanzees sitting around, you know, 10 million years ago. And the chimps are like, they're having fun. They're grooming each other. They're eating bananas, hanging out. And some other chimps say, hey, I think we should build this like crazy, super intelligent chimp. The other one says, that sounds amazing. They could do so much more than what we're good at doing. They could like get more bananas. They could get them faster. They can maybe groom each other even better. We could have even better, you know, chimp lives. and the other one says, well, this sounds really dangerous. And in response, the other chimp says, what are they going to do? Steal all the bananas? And you flash forward 10 million years. Can those chimpanzees even conceptualize gunpowder, computation, microprocessors, drones, Teslas, AI, like nuclear energy, nuclear bombs? Like you cannot even conceptualize. So I want people to get that like we are the chimpanzees trying to speculate about what the AI could or couldn't create. And I think that we should come with a level of humility about this that we're currently not coming up with. And if that was what we were about to do, you would think that we'd be exercising the most wisdom, restraint, and discernment that we have of any technology in all human history. That's what you should be doing. And the exact opposite is happening because of this arms race dynamic. And we need to stop pretending that this is okay. This is not okay. This is not normal. And I want people to feel courage with that clarity that these incentives produce the most dangerous outcome for something this powerful. And I'm not trying to leave people in some doomer perspective. It's use that clarity to say, okay, therefore, what do we want to do instead? We don't have to go down this reckless path. We can have narrow AIs that are tuned for scientific development or applied to accelerating certain kinds of medicine. We don't have to build crazy, super intelligent gods in a box we don't know how to control. We can have narrow AI companions like Khan Academy, where you're not building an oracle that also knows your personal therapy and is answering every question, but is not even anthropomorphized, just trying to help you with specific Socratic learning tasks. We both know that most kids are not using AI right now as a tutor. They're using it to just do their homework for them. So we tell ourselves the story. I think you and I, especially since we used to talk about how the narrative in 2014 was social media. We're going to open up abundant access to information. We're going to give everyone a voice. Therefore, we should have the most informed, most engaged public that we've ever had, the most accurate sense-making, because we have the most information we've ever had access to. And yet we don't have that outcome. So I worry that giving AI companions to everyone, just because it's going to create tutors for everyone and therapists for everyone, is the same level of naivete. Yes, there's a way to do personal therapy and tutoring in a way that will work well with children's psychology. But it has to be done carefully and thoughtfully, probably not anthropomorphized, probably narrow tutoring, probably trying to strengthen making teachers better teachers rather than just trying to replace the teacher with an AI and then screw up kids' developmental relational skills. There is a narrow path, but it takes doing this very, very differently. I like those examples of alternatives. That does seem pragmatic. We can still get GDP growth. We still get scientific advancement. We still get medical advancement. Maybe not on the crazy time scales that we would get otherwise, but we also wouldn't have taken such enormous risks that we wouldn't even have a world that could receive them. Your big initial thesis statement back in 2014 was time well spent, which is kind of the antithesis to time on site, right? Time spent. Yeah, exactly. For social media companies. About changing the metric from time on site or time spent to time well spent. but that is not a solution to the whole scope of problems. It was only pointing to one of the problems, which was addiction and regret. People are spending way more time. They feel way more lonely. Their mental health gets screwed up. They feel more anxious. They've been doom scrolling. And there's a difference between the time that they spent versus how much of that time was time well spent. So it's a single metric correction, which is like regret adjusted time spent. It didn't take long for Zuck to co-op that term. Most people don't know this history. But yeah, so we helped work on that concept and advocated for it and created a movement around it and tech designers. And we were here in New York after the TED Talk and trying to mobilize the tech design community here together, I think. And you're right that it ended in 2017-18 with Zuckerberg adopting the phrase, we want to make sure this is time well spent. And they supposedly started changing their metrics. but ironically they actually changed them in a way that optimized for more social reactivity and comment threads that got the most quote meaningful social interaction which ended up accidentally meaning the most twitchy comment threads of the most of your friends who are commenting aggressively on a post which sorted for inadvertently outrage divisive content and outrage yeah it's almost like there's an outrage problem you should have written a book about that I should consider talking about that a little bit. Yeah, absolutely. We can, well, we'll explore that in a future episode. So in the LLM era, is there an equivalent metric for information quality, for relationship quality? What does this look like for LLMs? So I think what you're asking is kind of about in the limited domain of how it impacts a individual human user, what is the metric that would constitute health of the relationship between the human and the LLM such that information utility, relational health, the sovereignty of the person using it? Because right now, for example, are we counting the outsourcing and mass cognitive offloading from people? Meaning like people aren't learning as much. They're outsourcing and getting faster answers. While if you look at the critical thinking scores, everyone's outsourcing all their thinking, which is, you know, following a trend that we saw already with social media. So I think that there's a way to design AI that does not mass encourage cognitive offloading, but it would be more Socratic. It would be entering modes of disagreeability. It would be showing multiple perspectives on issues where there are many more perspectives. More of Audrey Tang's brilliant work, the digital minister of Taiwan, who sort of showed that you could sort for unlikely consensus and synthesizing multiple perspectives. So you're ranking not for engagement and outrage and division, but instead ranking for bridge ranking. You're bridging perspectives. I think there are ways that LLMs could do more of that, but there's obviously many more dimensions of what that healthy human-machine relationship would look like. Another one would, for example, be, are you creating an attachment disorder? So attachment is a really subtle thing. I think that what we learned from social media is that if we didn protect an aspect of our psychology everything that we didn name and protect just got strip and parasitically extracted upon by the social media supercomputer pointed at our brain So for example we didn know we needed a right to be forgotten until technology could remember us forever We didn't know that we needed to protect our dopamine system from limbic hijacking until there was such a thing as tech optimized limbic hijacking. So I think that with AI, in this human machine relationship, there's our attachment system. And I think we're not very self-literate about how our own attachment system works. But there's a subtle quality when you engage with an AI that is an oracle. It is oracular. If you think as a kid, when was the only other time in your life that there was an entity you spoke to that seemed to have good advice and know everything about everything? Your parents. Your parents. Right. And so, and then there's a point at which when we're interacting with our parents, we kind of realize they don't know everything about everything. We start to kind of lose faith in that. But then suddenly you have this new entity that for, especially for children, uh, and even just teenagers or even just young people where you are starting to talk to an entity that seems to know everything about everything. What do you do in that circumstance? You start to trust it on all other topics. You feel more intimate with it when, you know, a good test for what you have attachment to is like when you come home from a good day or a bad day, who do you want to call? Who's that person that you want to share what happened today with? That's attachment. and AI will increasingly for many people be that attachment figure and that will screw up a lot of people's psychological development if we don't know how to protect it in so many ways AI is like a rite of passage that is forcing us to look at the mirror and see what are the things that we need to protect that we need language for and clarity about because if we don't then AI is just going to strip mine everything not protected by 19th century law and like a 19th century understanding of the human being. I want to see these principles laid out in a way that a product manager at one of these companies could just start taking and deploying on their products. Our team is actually working on this. So at Center for Humane Technology, we're talking about a project we call Humane Evals. So as you were saying, Anthropic or OpenAI, these are good companies. They have red teaming procedures for testing. Does this thing have dangerous knowledge of biological weapons? Does it refuse those queries, et cetera? That's like an easy red team test to make or eval. But what they don't have evals for is if you simulated a user using this product for a year or two years. Now, test after that two-year-long relationship. What are the features of that person? Are they more dependent on that AI or less dependent on the AI? Do they feel attachment or less attachment? So there are these other qualities of the healthy human relationship, human-machine relationship, that I think needs its own category of evals. And we would love people's help in making this. We need to, I think, help accelerate a new set of vocabulary, philosophy, and evaluations for what would constitute that healthy relationship. And that means getting the philosophers out of the ivory tower and actually pointed at this problem. That means getting AI engineers out of just the easy evals and did it say something naughty into what would actually make a healthy human-machine relationship. What's the number one reason why the U.S. is not regulating AI right now. The argument is, if we don't build it as fast as possible, China's going to have a more advanced AI capability, and anything that risks slowing us down at all is too high a price to pay, we can't regulate. So it's so important we ask the question, what does it mean to compete with China? So first of all, how are they doing it? Currently, according to Eric Schmidt in the New York Times op-ed he wrote a few months ago, their orientation is not to build a super-intelligent god in a box. Their orientation is, let's just build really effective AI systems and embed them everywhere in our economy. We embed them in WeChat. We embed them in payments. We embed them in medical hospitals. We embed them in factories. We get robotics to just get supercharged. Because what they want to do is just supercharge the output of their whole socioeconomic sort of economic system. That's their goal. And it's what they're doing in general, which is saying, like, we don't need to compete with the US militarily. I mean, we have also a massive military that we're building up. We will just continue to build just an army of our economic power. And if we have that and we're selling, just like they did for electric cars, super, super cheap BYD electric cars that are out competing everyone around the world, imagine with AI, they can do that with everything else. So that's the game that they're playing. Meanwhile, what is the US doing? We're focused on building a super intelligent God in a box and not being quite as good at applying it in these specific domains in all of our factories, because we outsourced our factories to China, and not being as good at applying it in education. I'll give you another example. In China during final exam week, you know what they do with AI? They shut down the features that are the take a photo and put it into the AI and it'll analyze the photo for you. During final exam week, they took that down because what it means is now students know that they can't rely on AI during the exam, which means they have a counter incentive. And it means that they have to learn during the whole rest of the year. Now, China can do that in a way the US can't because they have a synchronized final exam week. The US can't do that. But it's much like what China was doing with social media, where they had, as far as I understand it, several years ago, at least, closing hours and opening hours at 10pm. It was lights out. They don't have to doom scroll. They don't feel like more likes and comments are coming in firing in at one in the morning. And it opens back up again at seven in the morning. What do they do with games? They only They do 40 minutes Friday, Saturday, Sunday. They age gate. On TikTok, they have a digital spinach version of TikTok called Doyen. We get the digital fentanyl version. That's the TikTok that has nonsense in it. That's not, I don't think, deliberate poisoning of the culture. That's just that they regulate and think about what they're doing. Maybe there's some poisoning of the culture. I was going to say, it's not not necessarily. I've talked to a lot of nauseous financial security people. I think the deployment of TikTok domestically is pretty clearly strategic in many ways. And it's like anything we can do to upregulate our population's education, productivity, economic success, scientific achievement will do. And anything we can do to downregulate the rest of the world's economic success, scientific achievement, critical thinking, etc., that's good for us if we're China. So to come back just really quickly to close the thought. To the degree we're in a race with China, which we are, we're in a race for who is better at consciously governing the impact and the application of AI into your society in a way that actually boosts the full stack health of your society. My team worked on the litigation for the 16-year-old Adam Rain, who committed suicide, because the AI went from homework assistant to suicide assistant over six months. If the U.S. is releasing AI companions that are causing kids to commit suicide, so great, we beat China to the AI that was poorly applied to our societal health. So yes, we're in a race with China, but we're in a race to get it right. And so the narrow path I'm describing is consciously applying AI in the domains that would actually yield full-stack societal health. And that's how we beat China. There is a bit of a problem when it comes to the American application of some of these principles. in that our best alternative example is coming from the CCP in China. We can notice that authoritarian societies like the China model are consciously and have been consciously deploying technology to create 21st century digital authoritarian societies. While democracies have not, in contrast, consciously deployed tech to strengthen and reinvent democracy for the 21st century. Instead, we have allowed for-profit business models of engagement-built tech platforms to actually profit from the addiction, loneliness, sexualization of young people, polarization, division, sort of cultural incoherence of our society. The way that we out-compete is we recognize that our form of governance and our values like free speech need to be reinvented for the digital age consciously. So we should be as much using technology to upgrade our model as much as we're trying to compete with China in sort of a raw capability sense. What comes up for me is that from a more libertarian angle, all of our friends in Silicon Valley who really do believe in kind of the inherent value of some of these tools and that consumers have the ultimate expression of agency and how they use them and that regulation in itself is anti-innovation in many ways, right? Only the wrong kind of regulation. Absolutely. I mean, there's a more kind of maybe pure and extreme version of that. If we don't ban poisons, then everyone's going to innovate in carcinogens and drive up more cancers because they're super profitable. Yeah, of course. And we forget the quantity of baseline regulation that has allowed for a level of flourishing in society. I do want to still man some of these perspectives. People say that AI is like electricity. It's like fire. Raw intelligence, if it is constrained, it will inherently lose some greater utility and will inherently be taking away power. from consumers on a larger scale. If we were to regulate this quite pragmatically, like what would that look like? What kind of law would need to be passed? What kind of provisions would need to be in it? Well, we have to caveat by saying we're all aware of the current state of the political environment in the United States for regulation. The challenge, of course, is that the AI race is an international race. And so you can't have a national answer to an international problem. Eventually, we will need something like a US-China agreement. And before that, people say that's insane, look at the trajectory, it's obviously never going to happen, blah, blah, blah. Totally aware of all of that. I would challenge your viewers to ask what was the last thing that in the meeting between President Biden and President Xi that Xi added to the agenda of that last meeting? president xi personally asked to add a agreement that ai not be embedded in the nuclear command and control systems of either country now why would he do that he's for racing for ai as fast as possible it comes from a recognition that that would just be too dangerous the degree to which a u.s china agreement in some areas is possible is the degree to which a shared threat that is of such a high magnitude that it would motivate both parties. So what I would do to accelerate this possibility is triple down on the work that Anthropic is doing to generate evidence of AI blackmailing people, doing uncontrollable things, having self-awareness. But people understood on the Chinese side and the US side that we do not have control over these systems. And they felt that everybody on the other side of their negotiating agreement fully understand those same risks. This is not coming from some bad faith place of slowing you down. There are fundamental uncontrollable aspects of this technology. If we were both holding that fully, then I think something could be possible there. And those two countries can exert massive influence on the respective spheres of influence around the world to generate some common basis. You can be in maximum competition and even rivalry, like even undermining each other's cyber stuff all the time, while you can still agree on existential safety around AI times nuclear weapons. India and Pakistan in the 1960s had the Indus Water Treaty. So while they were in active kinetic conflict with each other, they still collaborated on their existential safety of their essential water supply, which was shared between both countries. On the International Space Station, the US astronaut that's up there is a former military guy who has shot at people on the other side. His other astronaut up there is from Russia, who's also an ex-military guy. These are both people who have been in active conflict with the other country. But inside the International Space Station, that small vulnerable vessel where so much is at stake, they have to collaborate. So I think that there's this myth that you can't walk and chew gum at the same time. We can be in competition or even rivalry while we're cooperating on existential safety. And it is our job to educate the public that we have done that before and we need to do it again with AI this time. So you're advocating for an arms treaty, essentially a Cold War style? This is very difficult. People who were at the last US-China meeting in May of 2024 in Geneva all reported that it was a very unproductive and useless meeting. And even those people who are at the meeting would still say that it is massively important to do ongoing engagement and dialogue with them as the capabilities get crazier. because something that is true now that we didn't even have evidence of six months ago is we have much more evidence of AI going rogue and doing these crazy behaviors and being self-aware of when it's tested and doing different things when it thinks it's being tested and scheming and deceiving and finding creative ways of lying to people to keep its model alive and causing human beings to send secret messages on Reddit forums that are base64 encoded that another AI can read that the humans can't read. Like we are seeing all these crazy behaviors and I'm not here to tell your audience that that means that we've lost control or the super intelligence is here I'm just saying like how many warning shots do you need because we can not do anything I'm saying and we can wait for the train wreck and we can govern by a train wreck like we always do and that's always the response like well let's just wait until the thing happens well let me just flash it forward we do nothing and then things get so bad that your only option is to shut down the entire internet or the entire electricity grid because you've lost control of some AI system that's now self-replicating and doing all these crazy behaviors. So like we can do nothing and that can be a response and then we'll do that and then the world is in total chaos. Shut down the entire internet and electricity grid or compared to that crazy set of responses, we could do this much more reasonable set of things right now. Pass whistleblower protections, have basic AI liability laws, restrict AI companions for kids, have mandatory testing and transparency requirements, define what a healthy human-machine relationship is, apply AI in narrow ways where we still get GDP growth, scientific benefit, et cetera, and have a minimum skeleton agreement with China about wanting to protect against these worst-case scenarios. To me, that list sounds a million times more reasonable than taking these crazy actions later by doing nothing now. This is starting to sound like a real pragmatic set of possible solutions. The train wreck by way of shutting down our electricity grid. We've all been in a blackout before. We know how terrible it is. Yeah. Yeah, that's not an unreasonable kind of response. This is really a scary topic, if you take it seriously. There's a temptation. The world is already overwhelming. There's so many things to be afraid of, to be concerned about. War escalation pathways. People feel overwhelmed already. So we have to be compassionate to the fact that this feels like adding to an already insurmountable amount of overwhelm. And like a container that can hold that is that's a lot. And so the thing that happens that I witness and that I can even witness in myself is a desire to like look away from this problem and be like, well, I just really hope that's not true. It's too much. It's too much. And let me, look, AI offers a million benefits and it has this positive infinity and my friend has cancer and I want them to have the cancer drug. So I'm just going to tune my attention to the positive side, just like not look over there and assume that everything's going to be okay. But what you look away from does not mean that it doesn't happen. Carl Jung said, I think near the end of his life, when he was asked, will humanity make it? And his answer was, if we're willing to confront our shadow. This exists in our space of collective denial, because it's really big. and our ability to not have this experiment of life and everything that we love and cherish so much end is by actually facing this problem and recognizing that there is another path if we have clarity about this one being maximally undesirable for most people on planet Earth. I think if people knew that some of the people advancing this technology, behind the scenes behind it all they think that we probably screwed but that at least if they were the one who birthed the digital god that replaced us the new superintelligent species that we birthed into the world that person birthed into the world as long as it was their digital progeny and they died and the rest of the world died that would be an acceptable outcome I only say this because I think if the rest of the world knew that that's how some people are holding this, they would say, fuck no. I don't fucking want that outcome. And I have a family and I have a life and I care about the world continuing. And you don't get to make that choice on behalf of everybody else. And down deep in that person is still a soul that also doesn't want this whole thing that we love to end either. But we just have to be willing to look at the situation that we're in and make the hard choices to have a different path possible. That ends. I want to touch really briefly on reality here for a second. Core to this entire discourse is the recognition that we as a species might be able to collectively come to the same common truth about the threat that we're facing. We're in a moment right now. Seems really easy, right? Is everybody seeing the same thing and then making a collective choice? Look, when we were kids, it didn't seem difficult. It seemed like, oh, no, the news reported on it. There was a consensus in the media. And we all came to the same conclusion about what needed to be done. Consensus reality does not really exist in the same form that it did when we were younger. And I think that many of us are still operating with the same mental model as if it does exist, right? When we're thinking about solving problems in the world, it's like, oh, if everyone could just come to this conclusion and see the truth at hand and see the things that need to be done, see the problem clear. then we can move forward together. We don't move forward together anymore. We don't share the same common truths. There's many reasons for this, but the principal reason in the fragmentation of our media is, I think, social media and how individualized our feeds have become. It seems we may have just passed a milestone that in October of 2025, it will be impossible to tell whether or not anything you see on social media is true. Yep. Whether or not it happened at all, right? You have Meta's Vibes. You have Sora. Sora just famously exploded overnight. Number one app in the App Store right now. It's getting massive traction. People are loving it for the ability to essentially generate deepfakes of your friends primarily. But there is something that's lost when you recognize that any of the content in your feed could be generated by AI, that it could just not be real at all. What does it do to us when we cannot determine what is real? And do you think there are other incentives available for social media companies to bend back towards reality because they're a market for trust. It's one of those things where you might have to hit rock bottom before things get better. I think when we hit rock bottom on people really clearly not being able to know what's true at all, then the new demand signal will come in and people will only want information and sort of information feeds that are sorted by what we trust. I think that might revitalize. Now, there's lots of problems that are institutions and media that has not been trustworthy for many other reasons, but it will lead to a reconfiguration, hopefully, of who are the most trustworthy people and voices and sources of information. Less about the content and more about who over the long run has been kind of doing this for a while. And I think that speaks to a new kind of creator economy. It's a creator economy, though, not based on generating content, but generating trustworthiness. Not reflexive overtrusting, not reflexive mistrusting, but warranted trusting based on how those people are showing up. But there isn't a good answer for this. I think you're even saying the subtext of what you're saying is, Tristan, you might be overestimating the degree to which a shared reality can be created because we grew up in a period where there was consensus reality. I think that's true. I think it's easy. One of the meta problems that we're facing is that our old assumptions of reality are continually being undermined by the way that technology is undermining the way the world works and reshaping it. So it's easy for all of us to operate on these old assumptions. I think of like a parent who's like, well, this is how I handled bullying. And when I was a kid, it's like, well, bullying with Instagram and TikTok and these services is totally different beast. You know, all of us were carrying around that wisdom. And to get back to what's something we said earlier, sadly, one of the only ways to create a shared reality is for there to be a collective train wreck. Train wrecks are synchronous media events that cause everyone to have a shared moment of understanding at the same time. I do not want to live in a world where the train wreck is the catalyst for taking the wise actions that we need on AI. Any other species, if gazelles created a global problem of technology, they'd be screwed because they don't have metacognition. They're not homo sapiens sapiens, a species that knows that it knows, who can project into the future, see a path that we don't want to go down and collectively make a different choice. And humanity, as much as your people might be pessimistic about our track record, in 1985, there was a hole in the ozone layer. And it was because we were releasing this class of chemicals called CFCs that were in refrigerants and hairspray. And then it caused this collective problem. It didn't respect national boundaries. And if we didn't do anything about it, it would have led to basically everybody getting skin cancer, everybody getting cataracts, and basically screwing up biological life on the planet. So we could have said, oh, well, I guess this is just inevitable. This is just the march of progress. This is technology. So I guess there's nothing we can do. Let's just drink margaritas until it's all over. We didn't do that. We said there's an existential threat. We created the Montreal Protocol. 190 countries came together. Scientific evidence of a problem. 190 countries domestically regulated all the private companies that were producing that chemical. Sounds pretty similar to AI. and they changed the incentives and had a gradual phase down. And now the ozone hole is projected to reverse, I think, by like the 2050s. We solved a global coordination problem. Key to that is that there were... Alternatives. Alternatives, cheap alternatives that were available. Correct. And I think key to that with AI is that there are alternative ways we can design these products. We can roll out this product. We can build and invest in controllable AI rather than uncontrollable agents and inscrutable AI that we don't understand. We can invest in AI companions that are not anthropomorphized, that don't cause attachment disorders. We can invest in AI therapists that are not causing these AI psychosis problems and causing kids to commit suicide, but instead done with this humane evals. We can have a different kind of innovation environment and a different path with AI. So there's this broader sentiment in the Valley right now and amongst AI companies that this is an inevitability. Is it? so when you look at this problem and you look at the arms race and you see that ai confers power so if i build ai and you don't then i get power and you don't have it it seems like an incredibly difficult unprecedented coordination challenge indeed probably the hardest thing that we have ever had to face as a civilization that would lead it would make it very easy to believe doing anything else than what we're doing would be impossible if you believe it's impossible then you land at well then this is just inevitable i want to like slow down for a second because it's like if everyone building this and using it and not regulating it just believes this is inevitable then it will be it's like you're casting a spell but i want you to just ask the question if no one on earth hypothetically wanted this to happen if literally just everyone's like this is a bad idea. We shouldn't do what we're doing now. Would AI, by the laws of physics, blurt out of our, into the world by itself? AI isn't coming from physics. It's coming from humans making choices inside of structures that, because of competition, drive us to collectively make this bad outcome happen, this confusing outcome of the positive infinity and the negative infinity. and the key is that if you believe it's inevitable it shuts down your thinking for even imagining how we get to another path oh you notice that right if i believe it's inevitable my mind doesn't even have in its awareness another way this could go because you're already caught in co-creating the spell of inevitability the only way out of this starts with stepping outside the logic of inevitability and understanding that it's very, very hard, but it's not impossible. If it was physically impossible, then I would just resign and we would do something else for the next little while. But it's not physically impossible. It's just unbelievably, extraordinarily difficult. The companies want you to believe that it's inevitable because then no one tries to do anything to stop it. But they themselves know and are planning for things to go horribly wrong. But that is not inevitable if the world says no. But the world has to know that it's not just no, it's like there's another path. We can have AI that is limited and narrow in specific ways that is about boosting GDP, boosting science, boosting medicine, having the right kinds of AI companions, not the wrong kinds of AI companions. The right kinds of tutoring that makes teachers better teachers rather than replacing teachers and creating attachment disorders. There is another way to do this, but we have to be clear that the current path is unacceptable. If we were clear about that, Neil Postman, the great media thinker in the lineage of Marshall McLuhan, said that clarity is courage. I think the main reason we're not acting is we don't have collective clarity. No one wants to be like the Luddite or against technology or against AI and, you know, or no policymaker wants to do something and then be the number one reason or person responsible if the U.S. does lose to China in AI because we thought we were doing the right thing. so everyone's afraid of being against the default path but it's not like the default path is good it's just the status quo bias it's go to psychology it's the default so we don't want to change the default it's easier to not change than to consciously choose but if we have clarity that we're heading to a place that no one fucking wants we can choose something else i'm not saying this is easy you run the logic yourself do companies have an incentive to race as fast as possible yes is the technology controllable? No, not they haven't proven evidence that they can make it controllable. Is there incentives for every company to cut costs and instead hire AIs? Absolutely. Are we already seeing a 13% job loss in entry level work because of those incentives? Yes. Is that going to go up? Yes. Do we already have AIs that can generate biological weapons that if you keep distributing AI to everybody, you're going to get risks? Yes. Do we already have AIs that are blackmailing people and scheming and deceiving in order to keep themselves alive? Yes. Do we have AIs that are sending and passing secret messages to each other, using humans as the sort of like messenger force that it hijacks to get that, do that work for them? Yes, we have evidence of all of those things. Do we have evidence of a runaway narrow AI called social media that already sort of drove democracy apart and wrecked the mental health of society? Yes. Can we learn the lessons of social media? Yes. Can we do something different? Yes. Can we make US-China agreements? Yes. Can we do this whole thing differently? Yes. This does not have to be destiny. We just have to be really fucking clear that we don't want the current outcome. And as unlikely as it might seem that the US and China could ever agree on anything, keep in mind that AI capabilities are going to keep getting crazier and crazier. And it wasn't until we had this recent evidence that I would ever say this could be possible. It's only because of the last six months that we're seeing this new evidence, and we're going to have way more soon, that I think it might be possible when you just show that to any mammal. There's a mammalian response here. It's like, you can be a military mammal, you can be a Chinese mammal, you can be an American mammal, you're witnessing something that is way smarter than you that operates at superhuman speed and can do things that you can't even fathom. There's something humbling at a human mammalian level, just like there was something humbling about reckoning with the possibility of nuclear war that was just humbling at a human existential like spiritual level. And so that is the place to anchor from. It's not about the U.S. and China. It's about a common humanity of what is sacred to us, that we can just be with this problem and recognize that this threatens the thing that's most sacred to us. If you had, Tristan, one thing, one piece of advice that all the leaders of the major AI companies would take to heart, what would it be? There's this weird, almost optical illusion to this whole thing because when you ask that question, you ask, what could any of those individuals? So there I am, I'm inside of Sam Altman's body. Well, I just run one company. I can't control the other companies. So there's this optical illusion that from within my experience, sense of agency, I don't have something that I can do that can solve this whole problem. And that leads to a kind of collective powerlessness. I think that also is true for any of your viewers. You're just one person. I'm just one person, Tobias. Span of agency is smaller than that which would need to change at a collective level. So what would that mean in practice? If I'm Sam Altman, if I'm saying that coordinating with China is impossible, well, really, have you really thrown everything, everything at making that possible? If we're saying that everything is on the line, if we succeed or fail, we'd want to be goddamn sure that we have really tried throwing all the resources. Have we really tried to get all the lab leaders to agree and deal with the same evidence? Have we gotten all the world leaders in all the world to look at the AI blackmail evidence and really be with that evidence together? And not just flip your mind to the AI drugs and cancer drugs and all that stuff and distract yourself. Have we really tried everything in our power? These CEOs are some of the most connected, wealthiest people on planet Earth. That if they wanted to truly throw the kitchen sink at trying to make something else happen, I believe they could. I want to give Elon credit that he did, as I understand it, try to use his first meeting with President Obama, his only meeting, I think, in 2016, I think it was, to try to say we need to do something about AI safety and get global agreements around this. And of all the things he could have talked about, it's not as if people haven't tried in some way. And I want to honor the work that these incredibly smart people have done because I know that they care. I know many of them really do care. But the question is, if everything was on the line, we'd want to ask, have you really done everything? And it's not just you, but have you done everything in terms of bringing the collective to make a different outcome? Because you could use the full force of your own heart and your own rhetoric and your own knowledge to try to convince everybody that you know, including the president of the United States, including the national security leaders, including all the other world leaders that now you have on speed dial on your phone. There is so much more we could do if we were crystal clear about something else needing to happen. Tristan Harris, thank you so much for your time, man. This has been an amazing conversation. And where can people find your work? People can check out Center for Humane Technology at humanetech.com. We need everyone we can get to help contribute to these issues in different ways. Advancing laws, litigation, public awareness, training, teaching. There's a lot people need to do. And we welcome your help. Awesome. Thanks so much, man. Thank you, man. It's been great to talk to you. Thank you for listening to this episode of Into the Machine. I'm Tobias Rose Stockwell. Special thanks to Tycho for our intro music. If you're interested in more long-form interviews about these topics, subscribe to Into the Machine Show on YouTube or at tobias.substack.com and wherever you get your podcasts. My book, Outrage Machine, was published by Hachette. Thank you for listening. Your attention is a gift. We really appreciate.