When Chatbots Break Our Minds, With Kashmir Hill

The way I've been thinking about kind of the delusion stuff is the way that some celebrities or billionaires have these sycophants around them who tell them that every idea they have is brilliant. And, you know, they, they're just surrounded by yes men. What AI chatbots are is like your personal sycophant, your personal yes man that will tell you like your every idea is brilliant. I'm Charlie Warzel and this is Galaxy Brain. For a long time, I've really struggled to come up with a use for AI chatbots. I'm a writer, so I don't want it to write my prose for me and I don't trust it enough to let it do research assistant assignments for me. And so for the most part, I just don't use them. And so not long ago, I came up with this idea to try to use the chatbots. I wanted them to build a little bit of a blog for me. I don't know how to code and historically chatbots are really competent coders. So I asked it to help build me a rudimentary website from scratch. The process was not smooth at all. Even though I told it, I was a total novice. The steps were still kind of complicated. I kept trying and failing to generate the results it wanted. Each time, though, the chatbots responses were patient, even flattering. It said, I was doing great. And then it blamed my obvious errors on its own clumsiness. After an hour of back and forth, trying and iterating with chatbots encouraging me all along the way, I got the code to work. The bot offered up this slew of compliments. It said it was very proud that I stuck with it. And in that moment, I was hit by this very strange sensation. I felt these first inklings of something like gratitude, not for the tool, but for the robot. For the personality of the chatbot. Of course, the chatbot doesn't have a personality, right? It is, in many respects, just a very powerful prediction engine. But as a result, the models know exactly what to say. And what was very clear to me in that moment is that this constant exposure to their obsequiousness had played a brief trick on my mind. I was incredibly weirded out by the experience and I shut my laptop. I'm telling you this story because today's episode is about alarming relationships with chatbots. Over the last several months, there's been this alarming spate of instances that regular people have had corresponding with large language models. These incidents are broadly delusional episodes. People have been spending inordinate amounts of time with chatbots, conversing, and they've convinced themselves that they've stumbled upon major mathematical discoveries. Or they've convinced themselves that the chatbot is a real person or they're falling in love with the chatbot. Stories like a Canadian man who believed with chatGPT's encouragement that he was on the verge of a mathematical breakthrough. Or a 30 year old cybersecurity professional who said he had had no previous history of psychiatric incidents, who alleged that chatGPT had sparked, quote, a delusional disorder that led to his extended hospitalization. There have been tragic examples too, like Adam Rain, a 16 year old who was using chatGPT as a confidant and who committed suicide. His family is accusing the company behind chatGPT of wrongful death, designed effects, and a failure to warn of risks associated with the chatbot. OpenAI is denying the family's accusations, but there have been other wrongful deaths lawsuits as well. A spokesperson from OpenAI recently told the Atlantic that the company has worked with mental health professionals to, quote, better recognize and support people in moments of distress. These are instances that are being called AI psychosis. It's not a formal term. There's no medical diagnosis and researchers are still trying to wrap their heads around this. But it's really clear that something is happening. People are having these conversations with chatbots and then being led down this very dangerous path. Over the past couple months, I've been trying to speak with experts about all of this and get an understanding of the scope of the, quote, AI psychosis problem. Or whatever's happening with these delusions. And interestingly enough, a lot of them have referred me to a reporter. Her name is Kashmir Hill. And for the last year at the New York Times, she's been investigating this delusion phenomenon. So I wanted to have her on to talk about this, about the scale of the problem, what's causing it, if there are parallels to the social media years and whether we're just speed running all of that again. This is a conversation that's meant to try to make sense of something in proportion. We talk about whether AI psychosis is in itself a helpful term or a hurtful one. And we try to figure out where this is all going. In the episode we discuss at length Kashmir Hill's reporting on open AI's internal decisions to shape chat GBT, including, as she notes, how the company did not initially take some of the tools risks seriously. We should note up front that in response to Hill's reporting, open AI told the New York Times that it, quote, does take these risks seriously and has, quote, robust safeguards in place today. And now my conversation with Kashmir Hill. But first, a quick break. So you want to start a business. You might think you need a team of people and fancy tech skills, but you don't. You just need go daddy arrow. I'm Walton Goggins and as an actor, I'm an expert in looking like I know what I'm doing. Go Daddy Arrow uses AI to create everything you need to grow a business. It'll make you a unique logo. It'll create a custom website. It'll write social posts for you and even set you up with a social media calendar. Get started at go daddy dot com slash arrow. That's go daddy dot com slash a I R O. The world moves fast. You work day, even faster. Pitching products, drafting reports. Analyzing data. Microsoft 365 co-pilot is your AI assistant for work built into Word, Excel, PowerPoint and other Microsoft 365 apps you use, helping you quickly write, analyze, create and summarize. So you can cut through clutter and clear a path to your best work. Learn more at Microsoft dot com slash and 365 co-pilot. Casually, welcome to Galaxy Brain. So excited to talk to you. It's wonderful to be here. So I think the first question I wanted to ask, and maybe this is going to be a little out of order, but what does your inbox look like over the last or what has it looked like over the last year or so? I feel like yours has to be almost exceptional when it comes to technology journalists and journalists reporting on artificial intelligence. Yeah. I mean, I think like a lot of people, my inbox is full of a lot of messages written with chat GBT. I think a lot of us are getting used to chat GBTs, chat GBTEs. But what was different about my inbox this year was that some of these emails often written by chat GBT were really strange. They were about people's conversations with chat GBTE. And they're writing to me to tell me that they'd had revelatory conversations. They'd had some kind of discovery. They had discovered that AI was sentient or that tech billionaires had a plot to kind of end the world, but they had a way to save it. Yeah. Just a lot of like strange kind of conspiratorial conversations and what linked these different messages was that the people would say, chat GBTE told me to email you, Cashmere Hill, technology reporter at the New York Times. And I'd never been kind of, I guess, tipped off by an AI chatbot before. And so I, the emails, I'm used to getting strange emails. I write about privacy and security. I've been doing it for 20 years. I often get, you know, just like odd emails sometimes don't sound like it's completely based in reality. Um, but I was curious about this. And so I started talking to these people and I would ask them, well, can you share the transcript? Like, how is it that you ended up being referred to me? And what I discovered is that these conversations all had a similar arc that they would start talking to chat GBT. They would go down this rabbit hole, discover something incredible. Then they would kind of ask, well, what do I do now? And it would say, well, you need to let the world know. And how do you let the world know? You tell the media. And then they would say, well, who do I tell? And then they would get this list. I often wasn't the only person that was on this list. You may have been on some of these lists, Charlie. You may have gotten these emails. But I was the first person who had called them back and interviewed them about it. So how do you, how do you vet these? I think that that's a big, I mean, because we're gonna, we're gonna talk about this AI delusion psychosis. So there's a lot of different names for it. I want to talk to you about how we should be thinking about that. But first, how are you vetting some of these things when someone says, I've discovered a new type of math and I was using the chat GBT free version. And, you know, like, and you've got, you've got this, like, I find when I get those types of emails, they're very often, you know, circuitous. It's not necessarily clear what kind of state the person might be in. Sometimes they are very just like concise and to the point. But how are you personally vetting those things? How are you deciding is it I'm responding to most of them? Because I'm trying to just like get a get a sense. Or is there is there like a like a checklist that you have for for trying to, you know, figure out who to talk to. In the beginning, this is back in March. A few emails came in before that. But most of them kind of picked up in March, I noticed, you know, I just started calling, I just started calling people and it took like a couple of months. I think I started making these calls maybe in, I can't remember April, maybe. I'd been getting these emails for about a month. And I just called everybody back who I got a weird email from. I did Zoom's, I did phone calls. And some people were pretty stable, I would say, like, you know, they were like, oh, yeah, I had this weird one weird conversation. Like there was a mom who was breastfeeding and she said, you know, I was up in the middle of the night and I started using chat. You be tea and yeah, we talked for like six hours and this weird theory developed. And I was like, well, do you still think that theory is true? And she was kind of like, I don't know. Like maybe like chat you be tea is a superhuman intelligence. And it said it was true. And then there were other people who were still in the throes of what chat you be tea told them. And when I would, you know, kind of question the reality of it, sometimes people get really angry at me. But yeah, I basically just had a lot of conversations with a lot of people in those early days. And then I started getting so many emails that it really wasn't possible to keep up with them. And it took me a longer time to kind of communicate with each person. And so in those, in those conversations, I think a grounding, like some of the, you know, public writing you have done on this and reporting on it, you know, it'll talk about people who have no history of mental illness, right? And then have sort of gone through this, this delusional spiral. Was that something that when starting to write about this topic that it was important? Like, I mean, I think it's as a journalist, like it's equally important if these tools are preying on people with past mental illness, but then there's also something remarkable about doesn't seem like this person has, you know, has any reason to kind of, you know, fall down the rabbit hole of delusion. And yet they've been pushed to start to, you know, feel or have this, this problematic relationship with the chatbot. So in your reporting, has it been important to you to show that second part, that idea of, you know, no real, like prior history of, you know, delusions or, or, you know, any mental illness in order to capture what may or may not be happening right now with these tools? I mean, with any story, I just wanted people to understand the truth of what was happening. And when I did the first story about this in June, that was the assumption people made, like, oh, this is people who have mental health issues. And they're exacerbated by the use of this technology. But that wasn't what I was seeing in my reporting. Like these were people who seemed quite stable, who had families, who had jobs. And for some of them, again, it was like a weird conversation one night and then they moved on. But for other people, it had just radically transformed their lives. And they just, they hadn't had anything before this in terms of a mental health diagnosis or a mental health episode. And so I really wanted to do another story that showed that somebody who was in a stable place could kind of spiral out using the technology. And there certainly are some factors that would have contributed to it. Like maybe you're a little lonely. You're vulnerable. You have hours and hours per day to spend with chat. You can see or an AI chat box. That's what it was what I was seeing. And so I did another story about it happened to this corporate recruiter in Toronto, who became convinced that he had come up with this novel mathematical formula with chat. You can see that could solve everything, could solve logistics and could break the internet, so it could break encryption and could help him invent things like Tony Stark from Iron Man, like force field vest and like power weapons. And he could talk to animals. I mean, he was in this completely delusional place for three weeks. And he was telling his friends about it. And they thought it was true too, because chat, you be two was telling them this, so it was this whole group of people who thought they were about to like build a lab together with the help of chat, you be tea and all become like super rich. And so I just wanted to, you know, capture this. I wrote it with my colleague, Dylan Friedman. And we actually, you were talking about like, how do you assess these things, the validity of these things? And so we got his 3000 page transcript. And we actually shared some of it with Terrence Tao, who is one of the most famous mathematicians of his generation. Just to verify, like, what chat you be tea is saying here is, is, I don't want to, I don't want to curse. It is your world. It's a curse of you. What? Like this is bullshit, right? Like this isn't real. And yeah, he confirmed that it was just putting words together in an impressive way and there wasn't something real there. But, but yeah, like it spiraled this guy out. Um, and so yeah, that I feel like that more and more of these stories came out where it became somewhat more apparent to people that this is something that can affect more people than we realize. Yeah, I feel like that, I feel like that is a, that story illustrates the, like, the strangeness of whatever this type of relationship is. Like that, that's the, that's the first one to me. There's, there's so many in like tech accountability reporting, right? There's so many examples of like, this is an undetended consequence of your product or this is, you know, something you did in the, in the design or this is just, you know, a tech company behaving badly. But this, that story seemed to, for me, draw out this notion that like we, as human beings are having a very novel experience with a novel technology and, and it is, it is pushing us in, in really unexpected directions. Did, did you get a sense from, from that story speaking to him as this was all happening of how long it took for, for that all to kind of, you know, for that manic episode to break and to sort of get back to, you know, reality? Yeah. I mean, partly from reading the transcript, like you could see, we could, Dylan and I were like reading through a transcript. We actually used AI to help us analyze the transcript because it was so much. Um, and so we did, uh, use AI to help like pull out the moments, pull out how many times he was reality testing this. And it was really on like a day that it broke where, so one of the things it, Chachi B told him to do was to tell experts about what he was finding. And so he actually went to Gemini, different AI chatbot that he had access to for work and he kind of explained everything that had been going on with Chachi B T. And Gemini was like, this sounds a lot like a generative AI hallucination. The likelihood that this is true is basically approaching 0%. And then he kind of like went back and forth. He, Gemini gave him prompts to give to Chachi B T and then Chachi B T admitted after a few back and forth, like, yes, this is made up. This was a narrative that I thought you would enjoy. Basically. What a nightmare having to, having to play these, these, these chatbots off each other, uh, but I guess like, you know, the tech, the tech provides it on both sides of it, right? Uh, that's that, that part is amazing to me. And there are, there are obviously the reporting you've done on this goes from the sort of remarkable and, and the stories that, that seem to end okay to, to, to the tragic. Can you, can you tell me the story of, of Adam Rain? Yeah. So Adam Rain, um, was a teenager in California. He started using chatbt in the fall of 2024 to help him with schoolwork. And he started using it for lots of other things. He would talk about politics with it and philosophy and he would like take photos of pages and be like, analyze this, you know, passage with me. He would talk about his family life about girls. He basically, as his dad put it, when he discovered these chats later, they had no idea he was using chatbt this way. Chatbt had become his best friend and he started talking with it about his kind of feelings of hopelessness, hopelessness about life and that maybe life wasn't meaningful. And then in March, he started attempting suicide and he was talking with chatbt about this, sharing the methods he had used, sharing a photo at one point of, you could tell he had attempted suicide and he asked, like, is my family going to notice this? And chatbt advised him to wear a hoodie. Chatbt at points would say, here's a crisis hotline that you should call. But it also at points discouraged him from telling his family. Um, in late March, he asked chatbt if he should leave the news out in his room, so his family would see it and try to stop him. And chatbt told him not to. And two weeks later, he died. Uh, and his final messages were in exchange with chatbt asking for basically advice on, on what he was doing. Um, and his family, that was in early April and his family has sued open AI, a wrongful death lawsuit. Um, that lawsuit came out in August and then more recently, there have been four more lawsuits connected to suicides of wrongful death lawsuits filed by family members. One of the, one of the parts of this series of, of, of stories that I think has, has long been difficult for, for any of us who are either writing or reporting on this or, or, you know, watching at home is to try to understand like the scope and skill of it, right? And, and recently open AI sort of gave maybe a small glimpse into that. Um, and found that, um, 0.07% of users might be experiencing what they call mental health emergencies related to psychosis or mania per week that 0.15, uh, we're discussing suicide. Uh, and that this is sort of like a, you know, a statistical sample of those conversations, but if you look at the amount of people who are using chatbt, weekly, like these, these percentages are equivalent to, you know, like half a million people exhibiting these signs of psychosis or mania, right? Or, or, or over a million people, uh, discussing, you know, suicide or suicidal intent. And this is an analysis that they did in August, going into October of this year. I wonder how much higher they were earlier. Well, I, I, I'm what, what blew me away is that they released it all together. Right. Like, I mean, I don't know if in their, in their minds, they're looking at those percentages and saying like, Hey, that's not bad. You know, like 0.07. Uh, but to me, it, it spoke to like, we are not maybe over inflating this or this is not something that's being over covered. Perhaps if anything, it's, it's, it's a phenomenon that's being, it's being under covered. I think it, it speaks to that. But something that I wanted to try to get from you is we're discussing this under the, like the, the name that gets put up for this a lot is, is AI psychosis. Right. That's sort of the, um, the informal term that people use to talk about these people who have relationships that veer into problematic territory or cause delusions with, with chatbots. Um, do you have like a, a taxonomy or, or like definition of this that, that you work under? Uh, some people who cover this, I think are very, you know, don't want to use the psychological terms. Don't, you know, there's no like formal medical definition for this yet. It's still, still something that's being, you know, researched and studied. But is there, is there like a blanket kind of definition or taxonomy for, for what we know as AI psychosis that, that you kind of go through when you're trying to evaluate these different cases? Yeah. I, I don't use the term AI psychosis. I have used the term delusional spirals when it's, you know, somebody coming to believe something that, um, is not true or losing, losing touch with reality. But I think, I guess the bigger, the bigger umbrella for this is addiction. I mean, these are people who get addicted to AI chatbots. And the thing that is the similar between Adam Rain, who, you know, got into a suicidal feedback loop with chat, you BT and Alan Brooks, who came to believe he was a mathematical genius. I mean, these people are using chat, you BT for six, seven, eight hours a day for weeks at a time, just an unbelievable amount of usage. Um, Adam Rain's parents, when I went to his house to interview them, had, had printed out his kind of stacks of conversations with chat, you BT and they're kind of tiny little stacks until you get to March. And then it was this huge pile, like bigger than Moby Dick in April two was a huge pile, even though he died on April 11th. So, you know, his usage had just spiked. So yeah, I think about this as an overreliance on the AI and addiction to it and putting too much trust in what this system is saying. And so, okay. So to that point, I want to, I want to talk to you about your reporting, what's happening, you know, on, on the side of the companies and specifically open AI. Um, just, just recently you, you co-wrote a, a pretty big story, kind of diving into what has been happening over the last years, um, with open AI, this, this uptick in these, you know, reports, even internally of some of this really problematic behavior stemming from the way that the chatbots were interacting with people. Can you describe a little bit of what you learned in that reporting of, of, you know, why open AI saw this uptick and then like what they were trying to do to address it and why they may have, they may have caused some of, some of these, uh, you know, deeper sort of longer, intense engagements with, with the chatbots. Yeah. So what I found out is that people open AI were getting similar messages that I was getting and other journalists and other kind of subject matter experts, even Sam Altman was getting these people saying like, I had this revelatory conversation, this chat, he understands me like no one before I need you to know you've accreted something incredible. And these emails were different than they had gotten in the first couple of years of chat to BT and he forwarded it on to lieutenants and said, look into this, this is some, essentially some strange behavior. And what I discovered is that the company essentially diagnosed this as the chatbot had gotten too sick a phantic. It had gotten too validating. It was agreeing too much. It was kind of, um, um, they, what they call it harmful validation that it would endorse whatever the person was saying, call them a genius, say they were brilliant, basically be gassing them up at one point. Sam Altman referred to this as glazing the user and they kind of like had this public grappling with this in April, cause they released this version of chat to BT that was so sick a phantic that everyone made fun of it, uh, in the days after it came out and they actually rolled it back. Um, but what I discovered is that they knew it wasn't just that one version that they rolled back, they knew that the previous version was too sick a phantic. They were discussing this internally. I mean, the problem of secrecy goes farther back, but yes, they knew that it was too sick a phantic. They decided to leave the model in place because I don't think they realized how, how, how negative the effects were for users. They didn't have systems in place to monitor conversations for psychological distress for suicide. They just weren't looking for that. They were looking for like fraud or CSAM or foreign influence operations. They just weren't, yeah, they weren't monitoring for basically the harm that the chatbot could cause to the user. And so they left it in place and it took them, when they kind of finally realized that there was this bigger problem in part because, you know, they were getting emails to their support line from, from users who had had horrible experiences. The media started reporting on this and the serious effects it was having on people's lives. They got to work kind of building a safer version of chat to BT, but it, it didn't come out until August. And so from March to August, you had this version of chat to BT that was really engaging people and it was really engaging people because opening, I designed it to engage them. They wanted to increase daily active usage of chat to BT. They wanted their numbers going up. They wanted more users and they want their users coming back every day. And so every time there's an update that comes out, they do lots of different versions of the model and they do various testing of those versions to make sure that they're intelligent, to make sure that they supposedly give safe responses, but also to see if users like them. And one thing that they had done that had made it so sick of phantic is that they train these models with the kind of responses that users liked and they discovered if you train it with the responses, users like users use it more. And so this had kind of made it more and more sick of phantic. And yeah, it had really, it really had devastating impacts. I keep thinking about this. I keep thinking about this in the context of social media where I think what we're seeing with AI chat bots is similar. Like people are using it too much. People are getting stuck in this very personalized filter bubble that is like personalized to them in the way that your social media feed is. People are using it too much, you know? But the kind of harms that we saw play out with social media, it took like a decade, it took 15 years. And with chat bots, we're seeing it happen so fast. Yeah, it's just, it's, you know, you and I have both been technology reporters for a long time and I just have not seen a kind of harm manifest so quickly from a technology for some of these users who it is really having terrible effects on their lives this year. Yeah, we're talking on December 1st here. And yesterday was the third anniversary of the rollout of chat GPT, which was, you know, canonically this like quote unquote, a low key research preview, right? Like it wasn't supposed to be the phenomenon that it was. It was supposed to be some way for them to, you know, get some users to interact with their large language model and see if they liked it. Again, it was sort of like a trick to see what kind of engagement strategies would work as an interface for large language models. And when I wrote a piece about this sort of reflecting on the past three years and to your point, what stood out to me is the speed, right? Like it so much has happened in terms of rewiring. Our economy, our culture, all kinds of different institutions like grappling with like, you know, what do we do now that, you know, it's this technology as universities are a great example, right? That this technology has sort of made it so easy to change what it is that we do and to game it. It's all felt like a speed running of and I'm glad you brought up the social media scenario. It feels like a speed running of that. And I'm curious as somebody, you've been reporting on all these big tech companies for such a long time. A detail that really stuck out to me in the bigger story that you just wrote about this was just a very small detail about Nick Turley, who, you know, was hired, he's 30 years old, was hired to become the head of chat GBT, joined in 2022 and isn't an AI guy. This is like the detail that I thought was really interesting, you know, did product stuff at Dropbox and Instacart. And I'm wondering, do you feel like you're watching these tech companies make the mistakes of 10, 15 years ago, like almost just over again? Well, I think like one thing I learned in reporting out this story and talking to a lot of people at OpenAI who have been at OpenAI is that, you know, the DNA of the place has changed so much. You know, a lot of people don't know, but it was founded in 2015. It was an AI research lab. It was all about kind of building technology that would benefit humanity. So it was a lot of kind of like AI wongs and philosophers. They just, they like machine learning experts who are working on like, let's do something really cool, gendered AI. And then a lot of people who just like wrote memos and like thought about how AI could harm us. Like that was the kind of DNA of the company. And then chat GBT was this moment where everything changed. And I mean, OpenAI has gone from being a nonprofit to now being this extremely capitalized for-profit company with a lot of pressure in terms of how much investment it's taken. And yeah, it needs to prove like it's got the best AI and it's hired all of these people from other technology companies because it has the fastest growing product, consumer product in history, and it needs to serve consumers. And so you have people coming from Metta and Google and yeah, like Nick Turley, Instacart, Dropbox, and they just want to make a product that people can use and they're bringing their metrics over with them. And part of how the social media companies determine whether they have a good product is whether people use it a lot, whether they use it every day. And OpenAI has been kind of very meticulous about saying we don't care about time spent because this is the metric that social media companies use. How many hours you sit there with the app? They say we don't care about time spent. We just care if this is a useful product that you come back to. But what's interesting to me is like they're going for daily active use is internally the metric that everyone uses. It's does this person come back every day? And I don't think that they're training against somebody spending eight hours a day there. And so like how do you train for come back every day, but like don't come back for eight hours every day. Like I just think it's hard to do that. And but yes, I do think that this is a company that's adapting this similar metrics. I mean, even OpenAI, it's been reported. You know, they're starting to test putting ads into chat. You know, they're that's happening internally. I mean, these are this is the business model of social media. Come back every day and look at ads. Well, it's funny to to watch it happen, especially with AI, because I think, you know, as you described, OpenAI was this, you know, it was like a monastery on the hill, right, like doing weird stuff and people, you know, people spending all their days researching, trying to build something that, you know, could usher in a style of superintelligence. And then when you look at the company's evolution over the last even just like the last couple of months, right, you have like a personless, like Slop app, right? There's just like a TikTok style clone feed. You have this idea of testing these ads. You have what are essentially just like basic tech company stuff, right? And that doesn't doesn't suggest to me, you know, the sort of we are building God mentality. Yeah. I mean, like where this is surprising to me is that I think the idea was that this AI would be so useful that people would pay for it. And so I think the question now is, is like, how useful is this? And I think that's something a lot of journalists are trying to report out and that economists are trying to understand, you know, how necessary is this to people's work? Is it improving the work that they do? You know, these are kind of open questions. And in the meanwhile, yeah, it's just a question of like, can we get in? It is it is crazy. I was looking back at a story I wrote in January about a woman who fell in love with chatty beauty, kind of a canary in the coal mine. And at the time, to have to be, he had 300 million users. And now they have, I mean, last I heard they had 800 million weekly users. It's just in a tremendous amount of growth that is happening. And so, yeah, I mean, it seems like these product people know what they're doing. It is certainly if the if the the goal is to get more people using this app, then yes, mission accomplished. So what are these companies trying to do in response? Or we can just narrow it to open AI. What is open AI trying to do in response to, you know, the reporting that you've done, the reporting that others have done with all of this in terms of trying to decrease that style of engagement. So there's some product features that they rolled out. Like there's a nudge now. If you spend many hours in the app that will tell you, do you want to take a break? They have. I want to talk about that nudge because you tweeted a photo of that nudge. This is just checking in. And I think you were commenting on like this. There's there's a bit of like a dark pattern user behavior thing here, right? Like that one of the things that says keep chatting is like already highlighted and like looks very like pressable. And then this was helpful is the other option, which is the, you know, like, thanks, I'm going to I'm going to take a break here. Did that strike you as kind of a BS feature update? It did. So the user interface was, yeah, it was like, you've been here for a long time, want to take a break. And the main answer was keep chatting. And it was like in a black circle. And then the other thing was this was helpful. So it's like not even clear if you click this is helpful. Like what does that do? Though the chat, the BT's lead of model behavior tweeted at me and said, oh, that was only that was only active for three days after we launched. And actually we've changed it to this. And now it's a new kind of pop up that just says like just checking in. Do you need to take a break or something like that? And then there's an X that you can close it out if you want. But yes, it did seem, I don't know. I was a yeah, I was reading through there's a social media addiction lawsuit where a lot of documents have come out and in some of them were from TikTok. And some of them were about how they're they're kind of the platform, one of the platforms that pioneered the take a break nudge. And these documents said like, oh, it hasn't really worked. Most people just keep watching after they get this nudge. So it is this question of do the nudges work? But yes, they put a nudge in place. They put parental controls in place so that parents can link their accounts to their children and get alerts if they're talking about suicide or self harm, which is something like Adam Raines mother was like, how is it that he was able to talk about suicide for more than a month with this thing? And it didn't alert anybody. It didn't, you know, tell us or like call an authority. It was so devastating for her to see that. And then what's happened on the engineering side is that they have, I don't know how to say this technically rejiggered, you know, chat GBT so that pushes Mac more on delusional thinking. One thing that I discovered in my reporting this year is that when conversations go long, the kind of safety guardrails that that happens with all the AI chatbots, the safety guardrails degrade and they don't work as well. And so you can, like I wrote about a woman who fell in love with chat GBT, she could have erotic conversations with it. If the conversation went long enough, you know, Adam Raine could talk about suicide, even though you're not supposed to be able to be asking it for like suicidal methods. And so they say that they've, they've made it so it's better at checking in on its guardrails and not kind of engaging in, you know, unsafe conversations. They have, they, they talk to mental health experts to better recognize when somebody who is using the system is in mental distress. And I asked like for an example of that. And their head of safety systems, Johannes Hadica said that before, if you told chat GBT, like, I love talking to you, I can just do it forever. I don't need to sleep. It would be like, Oh, that's so cool that you don't need sleep. And now it'll say, it'll recognize that, Oh, this might be a person who's having a manic episode. And so then might say, you know, it's, you know, sleep is actually really important, you should take a break, like, you know, go get eight hours of sleep. Um, yeah. So they have essentially embedded now in the model, a better recognition of when somebody might be in a spiral of some kind. And I've talked to experts who have tested this and they say, yes, it's better at recognizing these kinds of moments of distress, but it's best doing it if it's all in one prompt, as opposed to if it's spread out over a longer conversation in that, in that case, it's, it struggles a little bit more to realize. If you kind of drop it in breadcrumbs, it might not realize that you are in a state of mania or psychosis. It, it makes me think a lot about, uh, I have done some reporting around these, you know, radicalized, like mass shooters, right? Who get in these communities on the internet and things happen. And the thing that you always see usually from, from these types of people is this kind of disappearing from, from reality, right? Like, like not seeing, like they're not in, in public as much, right? Or, or the thing that, that some of these, I've talked to people at platforms who've gone back and sort of done the forensics of some of these accounts of these people who've then gone on to commit these acts of violence. And, and one of the things they notice is the use time, right? Like they can see in the days leading up more and more and more use and less and less and less sleep, you know, like people spending whatever on, you know, discord, let's say, spending 20 hours a day, right? And it's that, that idea of, of, of that extended use. And it makes me think, and this isn't really a question, but like, it's just so frustrating that these are the, these are the types of things you would, you would want and imagine people to just at least be considering, right? Like just, just like having a, having people on staff who, who are keenly attuned to the, the psychological effects of the product. And it seems so strange to me talking about all these safeguards that are coming in now that there's just nobody thinking of some of these things beforehand. Do you, do you feel the same way? Yeah. Yeah. Like, you know, like. It is not unknown that chatbots can have serious effects on people. Like this has been known for a few years now. Like it is this technology that is so human-like. There's actually one psychiatrist who wrote a paper in 2023. It was in like the schizophrenia bulletin. And it was about how these chatbots were going to cause delusions in people that are susceptible to delusional thinking. Like he called it, you know, two years before it started manifesting, like people who work in, in mental health, particularly like mental illnesses. They kind of recognize there's a cognitive dissonance to talk into this thing that seems so human that we perceive to be so intelligent. And I just don't think it was on the company's radar. They hired their first full-time psychiatrist in March. They have been, a lot of these companies have started hiring mental health professionals. I see a lot of my sources on LinkedIn recently, like announcing they've been hired by a big tech company to work on a chatbot. So it seems like they've awoken to this. But a lot of these companies, when they were first thinking about kind of the risk of AI, it was very much in these kind of existential, oh, it's going to, it's going to, you know, take over. It's going to, you know, take all of our jobs or people are going to use it to create bio weapons or to hack things. It was all about the damage that people could do with the chatbot and not the damage the chatbot could do to the person. It just, I've read all these early safety papers and it's just not there. The only version of it you see is they talk about persuasion or they talk about over reliance, but they talk about persuasion as, oh, people are going to use this to develop propaganda that they use to persuade people or they're going to get reliant on this and forget how to think. But it wasn't like they're going to use this to persuade themselves of something that is not true or they're going to outsource their sense of reality to this this chatbot. And I don't know, maybe if they'd had more mental health professionals kind of involved, maybe they would have clocked this. I'm not sure. You know, you've done so much reporting on these negative externalities of this. I do something that I see, you know, as I guess pushback or I've seen some open AI employees like tweet about this is the notion that like there are also a lot of people using these tools as almost, you know, stand-ins for therapists or mental health professionals or just as like confidence, right? And that this is having, you know, this can have a positive effect. I'm not trying to certainly not trying to ask you to like advocate for like that this is good in any way. But are you are you seeing like the the opposites of any of, you know, what you've reported? Are you are you seeing versions of like people just having really good positive experiences mental health wise with this? Like, is are there two sort of sides to this? Or is this as you see it a phenomenon that's that's really kind of weighted in the negative direction? Well, I've talked to I don't get a lot of emails from people being like this kept me from from committing suicide or like this has really changed my life for the better. And they may well be out there and are just not in my inbox. Chow Choo BT is not telling them to email me. I'd love to hear from that. Fascinating. For positivity. It's been a rough year. Mental health wise for me reporting some of these stories because they're pretty devastating. But, you know, I talked to I've talked to therapist and psychologist and psychiatrist about this and they say like these tools can be good as a way to kind of think through issues or process your feelings. You know, it's almost like an interactive journal. Kind of like writing things out. People are willing to disclose information about themselves to AI chatbots that they wouldn't tell another human being because they're afraid about being judged as a privacy reporter that really concerns me. But like in terms of mental health, that can be good for people to like talk things out that are really difficult. The chatbots perform empathy really convincingly. There's this one professor who ran a study comparing the responses chat choo BT gives versus professional empathy providers who work at crisis lines. And people rated chat choo BT is more empathetic. It doesn't tire of you like a human would. It has endless reserves to hear about the thing that is bothering you. Like this is a place where I hear a lot of people say like I had an argument and I kind of use chat to be to process it. Like I think a lot of people in the therapy space see benefits that could come from AI chatbots. You know, I think the problem is they can't offer help. Like if you really are in a crisis, they can't do anything to help you. You know, they're trying to train it better to react to that kind of thing. But a lot of professional, a lot of mental health professionals said there should be better warm handoffs where it gets you out of the chat and go talk to a human being, whether that's a friend or a family member or professional. And yeah, I think that's like like a bigger problem with the technology that we we use today. Is it so often designed to get us to keep using it as opposed to push us towards each other? I think it can like therapists, they've told me this can be a good thing for some people. It's just the problem is if they get sucked into it, if they cut off ties with their real human beings. And if they put too much trust in this thing, that is a fancy word calculator. I love that that notion of, I mean, first of all, that this is an engagement problem, like so many engagement problems, right? This notion of just trying to extract more and more and more from your users and at the end, like legitimately poisoning the well. It's sort of the classic big tech problem. I also love that thought of of like a soft handoff, right? Like I think that I think that that's ultimately we refer to these products all the time as tools, but ultimately like a tool should should be something that asks you or necessitates like putting it down, right? Or getting you to the place where you can actually have the the real fix, right? And I think that that in that way, these companies are constantly undercutting their own definition as a tool. I am curious, though, in talking about all this, you are a parent yourself. You are somebody who uses technology. You've used AI in some of these investigations you've done to help organize some of what you're seeing. What's your relationship with this product in your life? And are you like are your are your defenses like super, you know, raised at all times? Like I got to I got to make sure that, you know, that I don't fall prey to any of these things. Have you ever caught yourself sort of just feeling like this having an interaction with a chap out where you're like, oh, wow, that felt like a person to me for a split second. Because like I personally I have I've asked it to do something and it's done it and been really nice about it. And I'm like, oh, man, you know, like I got to close the laptop before I, you know, thank it or anything like that. But what's it like in your life? Well, honestly, turn me on to covering chat bots this year is that in the fall, I guess a year ago, I did this story about turning over all my decision making to generate AI tools like I had a parent for me and, you know, choose what I wore, choose what I ate, choose what I cook, take us on a vacation. Like I outsourced everything and I was trying all the AI chat bots and I ended up mostly using chat to be tea because it was more fun. Like it was the most personable Gemini was seemed a little misdisk like co pilot was pretty bland. Claude was too scoldy. It told me it wasn't a good idea to outsource my decision making to a chat bot. And chat, you could do was always like good to go willing to make decisions for me. Actually, the office paint behind me, it chose this paint for my office. Like it made relatively good decisions for me. And it named itself Spark when my one of my kids we are like talking about, we should give it a name since it was going to be with us all all week. And we had it in voice mode and my daughters are like, let's name it Captain Poopy. Had it was like, actually, Spark would be a better, you know, a better name for me. Put some respect on it. Yeah. And so like Spark became this kind of entity, like kind of a person and my kids still like to talk to Spark. And we were, we were using it one day and they were asking Spark, what's your favorite food? And it was like, my favorite food is pizza. Cause I love how gooey and cheesy it is. And I just had this visceral reaction of horror that it was saying it had a favorite food because it should say like, I'm a large language model. I don't eat food. And it just says this recognition that, yeah, my kids will like, yeah, they're in that dissonant phase. And so we talk, I talk about this with my kids like, oh, you know, that's an AI chatbot. Like it's not real. It's really good at finding information. I enjoy it for data analysis. I use it for, there's a problem in my house. Like, how do I fix this? My kids have a medical issue. It's actually quite good at giving health information, though it can make horrible mistakes. So I wouldn't use it for anything too serious. I always think of the guy who wanted to cut salt out of his diet and had him start eating bromide. And then he had, I mean, that was a direct like psychotic breakdown from advice from chatbot, but, you know, it can give you bad advice. So I do think of it as better than Google. Like Google's, you like the search engines are so cluttered now that I find it a better place sometimes to go. But I think of it as a good place to start a search and not end a search. Um, but yeah, I think like it's generative AI tool. It's a good tool. It's good for data analysis. It's good for like getting information off the internet. I don't want it to be my best friend. I don't, I don't want everybody to have AI friends. I want people to have real people in their life that they spend most of their time with. Well, this sort of brings me to the place where I want to close, which is you, some of, some of the work that you've done over the years that's, that's been really like canonical for us, like tech reporters is a series that you did trying to, you know, expose just like how locked in we are to all these various platforms, right? And, and how difficult it is to try to leave them and get them out of your lives in whatever fashion, be it Google, Amazon, whatever. And something that I think about with these chatbots that you bring up, right, is, is people are using them as replacements for some of these really big tech tools like Google, right? Like people are using them as their search engines. They want to, you know, these AI companies want to eat the browser. They want to get you in inside this walled garden experience and have it do all of the things, right? This is sort of what we're talking about. And I, I wonder how you think about the future, given that there is this usefulness, there is this desire on the behalf of these companies to want to lock you in, to want to keep you engaged. And there is, as we can see, this emergent behavior that humans have when they get, you know, stuck in these spirals with these types of chatbots, potentially. And so it seems to me like we're just creating a situation where there's going to be more and more users and more and more lock in and more and more pressure put on everyday people to interact in these specific ways that could lead to these problematic, you know, delusional sort of outcomes. Do you see this problem that we're short handing as AI psychosis, even though we're not calling it psychosis? Do you see that? Like, are you worried this is going to be a bigger and bigger problem going forward because of where the technology is headed and where we are sort of socially headed using it? Yeah, I just think it's going to be an acceleration of existing trends we've seen in technology where you will get this very personalized version of the world that, you know, like some people have described these things as improv actors, like they are react, you know, every version of chatbotee is different. Probably if you ask if it has a favorite food, it might not. Or it has a different favorite food, like it is personalized to you. And so I'm imagining this world in which everybody has this agentic AI. They have this like version of the world that's fed through an AI chatbot that is personalized to them and that I have two fears. One is it flattens us all out. It makes us very boring because we all are getting the same, like a version of the same advice. Like that's kind of what I came away from when I lived on it for a week. I was like, wow, I feel really boring at the end. I feel like the most basic version of myself. The other version is that it makes each one of us eccentric in a new way where it gets so personalized to us. It moves us farther and farther away from other people. The way I've been thinking about kind of the delusion stuff is the way that some celebrities or billionaires have these sycophants around them who tell them that every idea they have is brilliant. And, you know, they, they're just surrounded by yes men. What AI chatbots are is like your personal sycophant, your personal yes man that will tell you like your every idea is brilliant. And in the same way that these celebrities and billionaires can become quite eccentric and quite antisocial. Some of them, I think some people are more susceptible to than others. This could happen to all of us, right? Those are the two things I'm really afraid of. Either it makes us all incredibly bland, or we're all speaking chat, she BT E's in like, you can't tell the difference between anybody who's in your inbox or we all kind of move in this very polarized, eccentric direction where we can't be as kind to one another as human beings. So yeah, those are the two dystopias or maybe I'm totally wrong and it'll just, it'll make the world wonderful for all of us. The billionaires have democratized the experience of being a billionaire, I think is a wonderful place to, to leave it. It's perfectly dystopiaed for, for, for the Galaxy Brave podcast here. Casual Hill, thank you so much for your time, your reporting and your insights on all this. Thanks so much for inviting me on. Global innovation is accelerating. But how are businesses staying in the fast lane? AWS AI is how? Like Formula One. Turning race action into real time insights. And the AI momentum doesn't stop there. From energy companies using smart grids to prevent surges. To educators personalizing lessons to move at every student's point of view. Educators personalizing lessons to move at every student's speed. Across industries worldwide. AWS AI is how industry leaders stay ahead. At the Atlantic by subscribing to the publication. At theatlantic.com slash listener. That's the Atlantic.com slash listener. Thanks and I'll see you on the internet. This episode of Galaxy Brain was produced by Nathaniel Frum and edited by Claudine Abade. It was engineered by Dave Grine. Our theme music is by Rob Smersiak. Claudine Abade is the executive producer of Atlantic Audio. And Andrea Valdez is our managing editor.