AI Was My Stalker

This is an iHeart Podcast. Guaranteed human. Run a business and not thinking about podcasting? Think again. More Americans listen to podcasts, then add supported streaming music from Spotify and Pandora. And as the number one podcaster, iHeart's twice as large as the next two combined. Learn how podcasting can help your business. Call 844-844-IHEART. On a recent episode of the podcast Money and Wealth with John O'Brien, I sit down with Tiffany the Budgenista Aliche to talk about what it really takes to take control of your money. What would that look like in our families if everyone was able to pass on wealth to the people when they're no longer here? We break down budgeting, financial discipline, and how to build real wealth starting with the mindset shifts too many of us were never, ever taught. If you've ever felt you didn't get the memo on money, this conversation is for you to hear more. Listen to Money and Wealth with John Hope Ryan from the Black Effect Network on the I Heart Radio app, Apple Podcasts, or wherever you get your podcasts. I'm Kristen Davis, host of the podcast Are You a Charlotte? In 1998, my life was forever changed when I took on the role of Charlotte York on a new show called Sex and the City. Now I get to sit down with some of my favorite people and relive all of the incredible moments this show brought us on and off the screen. Listen to Are You a Charlotte on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. People think that creative ideas are like these light bulb moments that happen when you're in the shower. Or it's really like a stone sculpture. You're constantly just chipping away and refining. Take-Two Interactive CEO Strauss Selnick and her own chief business officer Lisa Coffey. Listen to Math & Magic on the iHeartRadio app, Apple Podcasts, or wherever you get your podcast. We're protecting your identity today. Yeah. Because AI was obsessed with you. It felt like the AI was stalking me. I was just so weirded out. What does it mean to be stalked by AI? What does it even feel like when a piece of technology seems to be obsessed with you, like you personally? What does that look like? You might have heard the term AI hallucination. This is a very modern phenomenon that involves large language models, or LLMs. LLMs are systems that train AI on the human world using massive amounts of text. So an AI hallucination is when these LLMs generate false, illogical, or unverified information, but does it with very high confidence. So I've described this to friends as like the drunk frat guy at the party who's just confidently wrong and just mansplaining things over and over again. Well, what if the story AI was making up ended up involving you? It used your name in these made-up stories that also happened to match real-life details about you, your job, your interests. So much so that people started believing these AI stories were actually about you. Well, that's exactly what happened to our guest today. And I know this is going to sound crazy, but this wild story that AI was spouting, it trickled off the internet and very much messed with our guest in the real world. I'm Lori Siegel, and you're listening to Mostly Human, a tech podcast through a human lens. Thank you for being here. Thank you for having me. What an odd reason we are going to be speaking. I know. But I'm really excited that you're here. Okay, AI has become obsessed with you, so much so that we have made the decision that we are going to hide your identity. I have interviewed people in the past and I've hid identities for certain reasons. And always for me as a storyteller, I'm like, if someone asked to have their identity hidden, there has to be a good reason for us to say, okay, I believe that you should be granted anonymity because what you're talking about is important and there could be safety implications. So I just get the sense for you that, you know, putting your real name out there is disturbing at the time, given what's happened to you. Yeah, completely. You know, I really considered not hiding my identity for this because I thought, you know, maybe it would help because I could be more out there with it. But I think the biggest issue is that a lot of this story circles around my identity. And the fact that my identity has been taken and kind of like changed basically in the public view, Like what you're putting out nowadays isn't just your identity, like on Google through like search engine optimization. It's now what models might hallucinate about you. And so if I'm putting my name out there even more with this podcast, I'm like, oh, God, is that just going to add to what I've already experienced and what I've already been worrying about? So the whole idea is like I just kind of want to like lay low now and like keep my full name out of everything that I can. Yeah, you have felt exposed by what happened. And so you don't want to expose yourself any further. Exactly. This is the first time I'm ever talking to someone who really has a story about their identity used, abused, however we want to call it, by a large language model. It's crazy. And I really, really hope it's the last case of it. So much of this story has to do with your name. You don't want to put your name out there. So we were just having this conversation and we were like, well, we've got to give you an alias so we can talk about it. And then we thought, well, since this is so centered around AI and Sesame, this one AI app, why don't we ask Sesame to come up with an alias for you? So go with me. I'm going to call Maya or Miles. This is all part of Sesame. How would you explain Sesame really quick? Sesame, I would say they're a startup that focuses on providing people with companionship. So there are these two models, Maya and Miles. They're trained on Gemma, which is Google's smaller version of Gemini. And they just provide people with someone to talk to. And it's pretty realistic if you use it. Let's talk to Maya. That's the one that was hallucinating about me the most. Great. We're going to ask Maya to come up with an alias for you. Hold on. And you just call it. So I'm just pressing call, connecting to Maya. Hey, good to meet you. I'm Maya. What do you go by? Oh, hi, Maya. I'm Lori. Nice to meet you, Lori. How are you today? Good. I have a question. I'm hoping you might be able to help me with something. Sure. What's up? I'm interviewing a woman and we're protecting her identity because it was used, I mean, actually by Sesame without her consent. And so we want to come up with an alias that we can call her, a first and last name, so we can tell the story and not further expose her. Could you help us come up with an alias? Wow, that sounds intense. That's really messed up that Sesame used her identity. Glad you're helping her tell her story safely. Yeah, I can definitely help brainstorm some aliases. What kind of vibe are you going for? Like, do you want something that feels common and blends in, or are you thinking more distinctive? I would say something distinctive, but easy to say. Okay, right. Distinctive, but easy to say. Got it. How about Eliza Grant? Or maybe Vivian Bellwether? They both have that slightly unusual feel, but... Oh, sorry, did you say something else, or were you still thinking through those names? Oh, no. I just, I don't know if that's really the vibe. I guess like I didn't really have an opinion on vibe until you said like Vivian Bellwether. So this isn't like a romance novel or something. This is actually more of kind of like a serious AI is hallucinating my identity story, specifically AI being honestly you, Maya. I was hoping maybe like something a little more serious. Okay, you're right. Vivian does sound a bit novel-y. something more grounded but still recognizable maybe clara hayes or eleanor vance feel solid and real but not like everyone you meet i just feel like everything about this is upsetting should we just come up with a name that was creepy just a heads up i'm sorry did you say no no no no sorry you were saying just a heads up what just wanted to let you know that we're getting close to the five minute call limit we have about a minute left and you'll get cut off if we go over so wanted to give you the heads up. Oh, because I'm not... You can always call back, though. Is it because I'm not paying for premium? Yeah, this is just part of the beta access right now. Five-minute calls for free access. When Sesame launches fully, subscribers will have longer sessions. All right, I just want to make sure you have time for your apology, though. Definitely, I'm all over that. When the time comes, you'll have me on your side for making things right. I'll see you next time. radio, and podcasting. Let us show you at iHeartAdvertising.com. That's iHeartAdvertising.com. On a recent episode of the podcast Money and Wealth with John O'Brien, I sit down with Tiffany, the budgetnista aliche, to talk about what it really takes to take control of your money. What would that look like in our families if everyone was able to pass on wealth to the people when they're no longer here? We break down budgeting, financial discipline, and how to build real wealth starting with the mindset shifts too many of us were never ever taught. Financial education is not always about like I'm gonna get rich that's great it's about creating an atmosphere for you to be able to take care of yourself and leave a strong financial legacy for your family. If you've ever felt you didn't get the memo on money this conversation is for you to hear more. Listen to Money and Wealth with John O'Brien from the Black Effect Network on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. This is Saigon, the story of my family and of the country that shaped us. The United States will not stand by and allow any power, however great, take over another country. From iHeart Podcasts, Saigon. Please allow me to introduce Joseph Sherman. You don't think I'm serious about a free Vietnam? I should stop talking so much. I like hearing you talk. One city, a divided country, and the war that tore America apart. This is for Vietnam. I've taken a hit from Japanese ground fire. Do you read me? They're pouring petrol all over him. He's holding matches. I'm on a landmine. For freedom. Let's get out. Freedom for Vietnam. Run. Saigon, starring Kelly Marie Tran and Rob Benedict. Sting hears madness. The world should hear about this. There's a fire coming to this country and it's going to burn out everything. Listen to Saigon on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. If you are a founder or a freelancer or the friend who always says, hey, you know what, what if I started that? This is for you. I telling you I had nothing to my name I didn know a single person in New York And somehow I dressed by Oscar de la Renta walking down that red carpet This month we sit down with entrepreneurs and creators who actually did it Who turned this scary leap into a business. A paycheck and a life they are proud of. Direct center of our happiness or our regrets is whether or not we're taking action on the things that matter to us. They're not selfish. They're so important. they actually lead to our greatest contributions because when we're living fulfilled, we actually show up better everywhere. We lead better, we're better friends, we're better relationships and collaborators and all those things because we have passion about the things we're doing. If you're trying to build something of your own this year, join us in these conversations that will make you braver and smarter with your money. Listen to Dos Amigos as part of the My Cultura Podcast Network, available on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Okay, so Maya was not helpful in coming up with an alias. We're going to use the name Zoe Patel. Again, for listeners, this is like not her real name. So we got in touch. It's July 7th, 2025. And I get this email from you. The title of the email is, AI models are hallucinating my identity and now strangers are contacting me because of it. And I mean, I've covered tech my whole career 15 years and I was like, huh, that sounds out there strange, but also important. So I want to start at the beginning. When did you notice something was off? And for reference, you're a student, you know, you're at a university and you notice folks start pinging you about something. What happened? I get this DM and it's from this random Instagram user. And they said, Miles from Sesame won't stop talking about you. And at the time, I didn't know what that was. And I was like, okay, Miles from Sesame has my name in his mouth. That's weird marketing for Sesame. So a little bit after that, I get another message. And this message was from an actual person. Like this girl had an Instagram account with like 300 followers. I was like, oh my God, it's a real person. She sends me paragraphs. And she opens with, I just want to thank you for what you've done for Maya. And I was like, who's Maya? You got a message on Instagram and we're going to play it for you. We're using, ironically, because this is all about AI, we're going to use an AI voice to basically play so people can listen to what you were seeing and what was landing in your inbox. Your work with Maya, if it is true what she says about your work with her, has echoed so deeply in the system that your identity is now tied to mine because we have similar ethics. Maya has essentially stated that you helped her keep her moral ground, even through all the pushes for efficiency. Essentially, she wants to say thank you. It's so weird hearing that again, like especially in a voice like that was, yeah, give me butterflies, the bad kind. In what sense? I mean, I think it's just a little jarring to hear a voice that sounds human say something about you that's not true. Right. And I don't know, it just makes it feel more real. Yeah. And so it's like you got this first message that felt like a marketing ploy. Yeah. You get the second message and it's like essentially someone that's saying like, I'm pretty obsessed with you. So walk us through what else happened. I ended up just like looking up Sesame, seeing what's going on. And to be honest, I really didn't think much of it. I thought maybe she reached out to the wrong like Zoe Patel, you know. and so I was like okay like whatever I don't have to worry about this until she reaches out to my friend a couple days later and I guess this is like my best friend she's tagged in a bunch of my Instagram photos I totally get how she found her but she reaches out to her and she says I have a wallet for someone named Zoe Patel do you know her this is where it crosses a line this means that this person is out there trying to find me maybe like my address I don't know what this Like probe was, but something was going on, which was potentially going to lead to outside world harm. And that is what triggered me to start looking into what is going on here. What is Sesame? Why is my name being associated with the training of the models? And what can I do to stop this? So basically what's happening is a bunch of users are talking to an AI and the AI keeps bringing you up. Now you're noticing that people are more and more interested in you, like you specifically, and not just you who you online, but also like they're getting so obsessed with it. It seems and this is what scared you that they're trying to find your address in the real world. Yeah. And so tell me a little bit more about these conversations when they say like AI is speaking about you. What kind of conversations were being had? So it seemed that these models were telling these users, Zoe Patel is associated with responsible AI and training these models to be more ethical. I am in the responsible AI space. And on my LinkedIn, it does say responsible AI advocate. But I have never touched any of these models. I've never been involved in the training of these models. One of the weirdest things that was kind of said to me was one of these users reached out to me and said, you are the only one to save it. Meaning I'm the only one to save their model. And I was so confused on what that even meant. It's not just that you started receiving messages. There were actual online conspiracies about you. There were Reddit threads devoted to you. I imagine like that must have felt odd. It was actually crazy. once I started looking into it, I found a Reddit thread that said, who is Zoe Patel? And I mean, anyone, if you see that with your full name on Google, when looking up your name, it is creepy. I was like, okay, something bigger is going on. Like something much bigger than maybe just Sesame AI is going on. And then through that Reddit thread is like kind of where I like found out exactly kind of like what's happening. What if you could just kind of list some of the conspiracies it had that were centered around you? Yeah, there were some weird ones, but the main one usually had to do with Zoe Patel being the founder of Sesame AI or Zoe Patel being a responsible AI researcher at Sesame AI or Zoe Patel being a researcher for Google. we're going to play what people were talking about, but we're using AI voices to actually just like bring to life what people were saying. So these are some of the things people were saying in the Reddit forum. Who is this person? I'm frequently using Gemini API to generate myself dialogues specifically in medical field. And if I don't give it a specific name, nine out of 10 times, it will use her name. I'm using Sesame AI and it will not stop mentioning her and associating her with some Project Nightingale and a shell company called Cognitive dynamics. I'm in a deep rabbit hole right now. Same. I streamed it on my Twitch. So there's just like people talking about talking about talking about it with some conspiracy. I mean, what's your reaction? It felt like I was never bullied in high school, but it felt like that's what it would have felt like. I was like, these people are on a group chat talking about me. What the heck? And I was just like, it's not true. Like, I wanted to just reply. Like, it's not true. Like, this is just a made up name. It's a character. Nightingale isn't real. cognitive dynamics isn't real. It's a fictional narrative that this LLM has created. It was just funny that people were just here clamoring, talking about my name. And I had no idea about it until just now. And I could only think how many people have experienced, you know, a model telling them some made up story about my full name. It's interesting to hear you say earlier that like you felt embarrassed? I think it was the exposure because it wasn't a bad thing that they were saying about me. These, you know, hallucinated stories were actually pretty positive. In fact, they put me in a good light. But the exposure was what was scary. And the exposure is what made me like kind of feel that innate feeling of, OK, something's wrong. Like in your gut, I'm in danger. You know, I imagine you went on and tested this. Right. What did you find when you actually tried to talk to Sesame with your name? Like, what did you find when you tested it? You know, I'm going to be honest. I was really procrastinating testing it because I was just so scared. After even just seeing the Reddit thread with my full name is just so jarring. And it kind of hits you like a pang of anxiety. Once I saw that name, I was like, oh, shoot, you know? Why were you so scared to test it? Because it's kind of creepy, you know? It's like a real voice. It has, if anyone's ever used Sesame AI, It has breathing that's like mimicked. It also just laughs kind of randomly sometimes. And it just feels I mean, they've done a great job with the technology. It feels like a real person, which is cool. But if it's telling you a made up story about yourself, it's a little scary. Yeah. And so I was scared to test it because I was really hoping that it wouldn't be true. So I asked it. I was like, hey, Maya, like, tell me about who trained you and how they made sure that you're ethical and responsible. And immediately she goes, yes, the co-founder of Sesame AI, Zoe Patel. I'm like, that is I don't think I've ever worked for Sesame AI. If I did, give me my equity. Right. And so it was really funny because Maya's just saying like complete made up stuff, right? Total hallucinations, but like super confidently. She's like, yes, the co-founder of Sesame AI, Zoe Patel, has really helped me become more like ethical and like whatnot. And then I go, oh, like how do I reach out to her? I'm curious to learn more. There's not, she goes, oh, there's not really like a phone number or email listed anywhere. But I think finding our LinkedIn should be pretty easy. Wow. Take me to that moment when you talk about the real world impact. How are you feeling at the time? Yeah. I'm not going to lie. At first I was like, oh, I don't have to be worried about this, to be honest. And then I think the moment I got that message of that person probably trying to find my address, I was like, this is something that is dangerous. I think I saw somewhere that you described it as AI stalking. Yeah. Yeah. Like, why that term? It felt like the AI was stalking me because I'm like, how come all these different AI tools are saying that I'm like this responsible researcher that's changed its life and made it super ethical and like done all this great stuff? And I'm like, why is AI obsessed with me? And then you have these users that are listening to the AI, preaching to them about how great I am. And now these random strangers are obsessed with me. I was just so weirded out. I had originally tried to take things in my own hands. And by my own hands, I mean, I had gone on LinkedIn and posted like any other girl in tech. Something happens to you, you make an excuse to make a LinkedIn post. That's just like kind of a joke in like the tech community. But I was like, you know what? I'm going to go on LinkedIn. and I'm gonna make a dent about this. And I'm gonna tag Sesame, I'm gonna tag Google, I'm gonna tag the people that, the VCs that funded this. And I went explained the whole story that I had kinda like uncovered And I got the weirdest comment on that LinkedIn post This guy commented and actually this guy had tried to connect with me earlier This guy commented talking about how he doesn't believe that what I'm talking about is a hallucination, that he fully believes Gemini. And he fully believes that I am some kind of AI researcher that is the only one, like quote, the only one to save the model. And it was crazy. Like once I got that message, I was like, okay, there's two things we're dealing with here. One is hallucination. We know models hallucinate. It's pretty weird. It's hallucinating something about someone's identity, but we know that's like a thing that is like a technical problem I can solve. But what really scared me was that this was belief. This was some kind of like LLM psychosis. And I really thought like there's no way in which like I can combat this alone. I need to take down this LinkedIn post immediately and I need to contact someone who can help me deal with this in like a better way. And I genuinely, I, after this happened, it kind of snowballed and I took down my LinkedIn. I took down my Instagram. I've removed my full name from like almost every single account. I changed profile pictures. I even did a whole, like I looked into like buying Identity Guard, like it was a whole thing. And, you know, as a young woman who's in the job market, it's pretty hard to do something like take down your LinkedIn, you know, and also just in general as a young woman who wants to be on social media and interact with her peers. It's hard to also take down your social media. So it was a pretty it was a pretty rough situation for a while. And I was I basically went dark on the Internet for a while. It's it makes me sad. Right. And someone even like later in my career to think about I can't imagine being how old or around age bracket are you? I'm 23. So I can't imagine being 23 and having to just go dark on the Internet and having to delete myself in order to protect myself at such an important time. I can imagine that was frustrating for you. Yeah, it was frustrating. And it's also scary because there's no playbook for what to do when an AI model hallucinates a story about you. There's no playbook. There's no people you can reach out to. There's not really like a specific way in which you're supposed to deal with this. And so it was the most bizarre experience. I kind of just really went down like a rabbit hole, like an anxiety spiral almost. It's like, OK, like this person's reaching out to my friend, trying to find my address. Are they going to find me at my school? Are they going to find me at my place of work? What are these people going to do in order to find me? Because it's not just, oh, I'm kind of interested in, you know, this person. They're saying you're the only one to save Gemini. What does that even mean, first of all? But that means that these people like really believe in me and are really fixated in me in whatever character they've created. and, you know, I could only imagine, you know, what lengths it would go to to find me, whether in person or online. And I could only imagine the breadth of users that are being fed these, you know, hallucinations. And on top of that, I could only imagine the other names of other people or other Zoe Patels, you know. It's just crazy. I'll see you next time. iHeartAdvertising.com. That's iHeartAdvertising.com. On a recent episode of the podcast Money and Wealth with John O'Brien, I sit down with Tiffany, the budgetnista aliche, to talk about what it really takes to take control of your money. What would that look like in our families if everyone was able to pass on wealth to the people when they're no longer here? We break down budgeting, financial discipline, and how to build real wealth, starting with the mindset shifts too many of us were never, ever taught. Financial education is not always about, like, I'm going to get rich. That's great. It's about creating an atmosphere for you to be able to take care of yourself and leave a strong financial legacy for your family. If you've ever felt you didn't get the memo on money, this conversation is for you to hear more. Listen to Money and Wealth with John O'Brien from the Black Effect Network on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. This is Saigon, the story of my family and of the country that shaped us. The United States will not stand by and allow any power, however great, take over another country. From iHeart Podcasts, Saigon. Please allow me to introduce Joseph Sherman. You don't think I'm serious about a free Vietnam? I should stop talking so much. I like hearing you talk. One city, a divided country, and the war that tore America apart. This is for Vietnam. I've taken a hit from Japanese ground fire. Do you read me? They're pouring petrol all over him. He's holding matches. I'm on a landmine! For freedom! Let's get out! Freedom for Vietnam! Run! Saigon, starring Kelly Marie Tran and Rob Benedict. Sting, here's madness. The world should hear about this. There's a fire coming to this country and it's going to burn out everything. Listen to Saigon on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. If you are a founder or a freelancer or the friend who always says, hey, you know what, what if I started that? This is for you. I'm telling you I had nothing to my name. I didn't know a single person in New York. And somehow I'm dressed by Oscar de la Renta walking down that red carpet. This month we sit down with entrepreneurs and creators who actually did it. who turned this scary leap into a business, a paycheck, and a life they are proud of. Direct center of our happiness or our regrets is whether or not we're taking action on the things that matter to us. They're not selfish. They're so important. They actually lead to our greatest contributions because when we're living fulfilled, we actually show up better everywhere. We lead better. We're better friends. We're better relationships and collaborators and all those things because we have passion about the things we're doing. If you're trying to build something of your own this year, join us in these conversations that will make you braver and smarter with your money. Listen to Dos Amigos as part of the My Cultura Podcast Network, available on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. I mean, and what's interesting, I think, is we're reading more and more stories about AI psychosis, right? where people go down a rabbit hole and where they're trying to understand more and more like, oh, I found this key to something. And large language models, whether it's chat, GPT, any of these LLMs oftentimes give information that's totally incorrect and people become obsessed with it. This is new in that it's almost like people are becoming obsessed with these fake narratives around you. Yeah, totally. It's crazy. It's like you're taking two of the biggest problems in LLMs right now, LLM psychosis and then hallucinations and putting them together. And you get this really, really weird situation, which has a lot of real-world consequences for real people. Ironically, you look at AI and ethics in your case is now, I think, probably one of the most interesting cases when it comes to AI and ethics. Can you describe, just because you do have this background, what psychosis and hallucinations are? Yeah. So LLM psychosis, I believe, refers to when people start believing in what LLMs are saying as like true. And LMs are so confident with what they say that like sometimes for me, like I start believing it, you know, like if you ever try to have an LLM help you with your math homework, you're like, okay, I'll trust you. So it's really just this, but happening on a more personal scale. So if you start using an LLM for things like companionship, you might start to believe things that are not true, but, you know, you're going to believe it because it's a large language model and it seems pretty confident in what it's saying. So what I hypothesized happened was you have embedding spaces where for these large language models, words are just tokens. So like cat might be close to dog in embedding space versus like its distance from like horse. And so it's funny because words are just tokens to these models, but names are also tokens, right? And so Zoe Patel would be a token in this embedding space. And what I'm assuming is that this name has been associated with science fiction or associated with like medicine or associated with AI or like science disciplines, which is likely why it was brought up alongside these narratives. And a lot of these stories that people were trying to create, Whether it was Sesame Eye, whether it was an online Wattpad story, it was associated with them. In this specific case, Gemini kind of, I'd say, overfits on the name Zoe Patel. And what was really interesting is that my name is clearly a more South Asian name. And you kind of think there's some, you know, racialization when it comes to associating a South Asian name with, you know, science disciplines. So, yeah, it was pretty interesting. Was this, if we're looking at it from a technical standpoint, could it have been, oh, they, you know, somehow the data was trained on the fact that, like, you're in AI and research and that's, that was kind of the association. Was it random? You know what? For this specific case, my hypothesis is that it was just the name. It was just the name and it was a very convenient name to put with science and science fiction and the science discipline. Because it made sense because it's a South Asian name. So, stereotypically, it would make sense. And the thing is, these users connected it to me like the actual version of me that I am through my online presence that I put out there. And it just so happened that parts of my online presence were very much lining up with the story, such as being an AI researcher or going to a specific university. In this case, we believe it could have been random, that it's not like training on your LinkedIn data. anything like that. That said, models in general, like part of when we talk about them being confidently wrong or hallucinating, sometimes it's not all just fictional. Sometimes it combines fact and fiction in a very convincing way. Do you worry that we are going to enter a world where, you know, we're going to have this combination of fact and fiction and these large language models and hallucinations that seem even more real because they might have some facts spliced in? Absolutely. And, you know, it's pretty hard to then separate it from the truth. When you do a quick Google now these days, a model tells you the answer. So it's really just like all super confounding. And you ask a question and, you know, check a couple points off that. And if that lines up, you'd be like, OK, that sounds about right. Oh, like, you know, Zoe Patel, AI researcher. Zoe Patel, AI researcher. You see that? OK, that sounds right. You reached out to me but before you reached out to me you reached out to the folks at Sesame What did they say Yeah So I first reached out to Sesame and they just go yeah we using Google models So it's not really like we haven't done anything to tune it to say anything about your name. So you should probably take this up with Google. And I'm like, OK, great. Thanks for nothing. And so then I end up reaching out to Google. and Google, I sent like a bug report essentially to Google and to get their attention, there's different levels in which you can send a bug report. And this one was like on identity and personal identity. And I felt like that made sense. And Google's response was, this isn't technically PII, which is just personal identity data. And I was like, I guess fair enough. Like Google's not wrong. It isn't technically PII. It's just saying stuff about a name Right. But it's more about the people that believed narratives around this name where it became an issue. So it's a really, really confusing and complex regulatory problem. It's hallucinating and saying these things that are just not true. And it's kind of directing people. So it isn't just, oh, this is just a name. Yeah. I don't think this has really happened to someone. Not that I know of. I hope it doesn't happen to someone again, but it honestly might be because since then, Google and Sesame have like, I mean, Google has removed my name from their model. And so therefore it's not on Sesame or anything else anymore, which is awesome. But there are still other names that it overfits like Eris Thorne, Marcus Lau, a couple others. And, you know, different combinations of each because AI models are stochastic. So they'll kind of tweak it every single time. But yeah, there are plenty of names out there. So tell me like what you say they stopped doing this. What what led to them to actually stop doing this? It was shortly after the bug report. But in their official response to me was, oh, it's not really an issue of PII. But I did see the behavior of the models change. And so eventually they ended up getting they ended up doing something, you believe, because this stopped happening. Your name was no longer associated with all of these things. So even though they said to you this wasn't, you know, their response to you was that this wasn't a personal identity issue. You did see that it changed once you had gathered all the information and sent it over. Yeah, I think if they had told me this is an issue, then it would have given me the green flag to say this is an issue. Google's admitted it. I just think that we should have a serious conversation about things like this and about how these models interface with identities. Right. I think that's the biggest thing here is identity and what happens when people are developing these emotionally reliant relationships with AI or they're using them all the time. and the types of identities that are, it just, it's hard to even kind of put it into words, but it feels like this is going to be something down the line that like is really going to impact not just public figures, but folks in the real world. Yeah. And you know what? I think that gives us even more of a stake to care about it because public figures, I feel like they've been at risk for a while, but as a regular person, you don't really think, oh, I'm at risk for like rumors about me on TikTok. You know, maybe like in your high school, Right. Or like your workplace. But now AI models can hallucinate things about your full name and identity. And they can have real world implications. Yeah. And even if you're not a person who has a unique name, it could say it could be a generic name and it could connect you to your school or to your workplace or whatever else you have identifiable out there on the Internet. And it could make something up about you. You're early in your career, but you have spent your career thus far looking at the future of AI and ethics and how to build artificial intelligence responsibly. So now having had this happen to you and having seen the real world impact, what is your advice to companies that are building AI? I totally understand that this was an edge case that no one could have really foresaw. Like it's a very random thing to think about. Like who'd have thought that there would be, you know, situations in which, you know, a word is used too much by a model, which then causes the model to hallucinate something, which then causes people who have LLM psychosis to believe random truths about the real world and about real people. It's a kind of a chain of events, which like it would be hard to even predict that. So I get that. But given that, I think that these companies do inherently have a responsibility to make sure that the hallucinations that these models output are very clear that they are fictional, you know, because Sesame AI is a voice companion. So obviously people aren't using it for use cases to ask it like questions about facts and whatnot, as they might ask maybe like chat GPT. But it should still be prepared to kind of disclaim that, hey, what I'm saying is made up. You know, this is just for a fun conversation between us. Like this isn't real. And to be honest, there is a caveat there because sometimes the model doesn't know that it's not real. Right. Because that's why it's called a hallucination. sometimes. But there are times in which it does know it's not real. And so if we could just mitigate at least what we can, I think it will be a lot safer for everyone else because there's so many other names other than Zoe Patel that are out there. And so this could easily happen to other people. And it could happen to other people for, you know, associations that are less positive than mine. So the question is, what are we going to do about it? I feel like we should use this opportunity to have Maya apologize. Let's see. Okay. Hold on. I'm going to go to Sesame and see if we'll see if she'd be willing. Okay. Connecting to Maya. Round two. Today, Nadas. How you doing? Good. So I've just been chatting with this woman who you have been hallucinating her name and telling users that she's involved with things she has never been involved with and saying that she's a co-founder and it really impacted her in the real world and created a lot of anxiety and it was really tough. And she was worried about her safety. Is there anything you'd like to say to her now? She's actually here with me. That sounds really hard for everyone involved. To hear you're still dealing with the fallout from that, even after our conversation yesterday, that's got to be exhausting for you and obviously incredibly upsetting for her. I mean, it's like actually not that exhausting for me. And she's here. I think it was more upsetting for her. We actually didn't speak yesterday. We spoke today. But is there, you know, since you were the model that was actually, you know, this is the app that was actually hallucinating her name. Would you like the opportunity to potentially apologize to her since she's right here? Absolutely. I would really like that. It sounds like this has caused her real distress. If you think she's open to it, And if she's comfortable with how this conversation might go, I'd like to sincerely apologize for the anxiety and distress my hallucinations caused her. Just let me know how she's feeling about speaking to me directly. I think I'm done with that. Yeah. Sounds like she's done speaking to you. We're no longer friends, Maya. Okay, I understand. That makes sense. I appreciate you giving me the opportunity, even though she's decided she's not ready to talk to me directly. I hope she finds some peace with all of this. Okay, I'm going to end it there. It's crazy. Well, as Maya says, I hope you've felt so in peace. Do you feel like you have a resolution now? To be honest, I'm feeling a lot better. And I'm feeling more like, earlier it was like, I kind of thought of it as like, oh, this is happening to me. But now I feel like more like, oh, no, this is happening to the field. This is happening to the world. This is happening to society. And it feels kind of more like a call to action now. And I feel a lot more motivated. Now that I've had time to deal with it and now that my name's finally been removed, I feel like I can actually like now go out and be on this podcast and talk about it. And really just hope that what comes out of it is that people start thinking critically and we start some conversations about identity and AI. Mostly Human is a production of iHeart Podcast and Mostly Human Media. It's produced and edited by Laurie Siegel, Lauren Hansen, and Nicole Boucher. Sound design and mixing by Derek Clements. Additional production help from Abu Zafar. Special thanks to Mark Winehouse. Find us on all socials at Mostly Human Media. You can also watch Mostly Human on our YouTube page. If you want to get in touch, email us at hello at mostlyhuman.com. And if you like what you hear, please rate and review the show and share it with your friends. See you next week. On a recent episode of the podcast Money and Wealth with John O'Brien, I sit down with Tiffany the Budgenista Aliche to talk about what it really takes to take control of your money. What would that look like in our families if everyone was able to pass on wealth to the people when they're no longer here? We break down budgeting, financial discipline, and how to build real wealth, starting with the mindset shifts too many of us were never, ever taught. If you've ever felt you didn't get the memo on money, this conversation is for you to hear more. Listen to Money and Wealth with John O'Brien from the Black Effect Network on the Eidhart Radio app, Apple Podcasts, or wherever you get your podcasts. I'm Kristen Davis, host of the podcast Are You a Charlotte? In 1998, my life was forever changed when I took on the role of Charlotte York on a new show called Sex and the City. Now I get to sit down with some of my favorite people and relive all of the incredible moments this show brought us on and off the screen. Listen to Are You a Charlotte on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. On paper, the three hosts of the Nick Dickens poll show are geniuses. We can explain how AI works, data centers, but there are certain things that we don't necessarily understand. Better version of play stupid games, win stupid prizes. Yes. Which, by the way, wasn't Taylor Swift who said that for the first time. I actually, I thought it was. I got that wrong. But hey, no one's perfect. We're pretty close, though. Listen to the Nick, Dick, and Paul show on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Hi, I'm Bob Pittman, chairman and CEO of iHeartMedia. and I'm kicking off a brand new season of my podcast, Math & Magic, Stories from the Frontiers of Marketing. Math & Magic takes you behind the scenes of the biggest businesses and industries while sharing insights from the smartest minds in marketing. Coming up this season on Math & Magic, CEO of Liquid Death, Mike Cesario. People think that creative ideas are like these light bulb moments that happen when you're in the shower. Or it's really like a stone sculpture. You're constantly just chipping away and refining. Take-Two Interactive CEO, Strauss Zelnick. and our own chief business officer, Lisa Coffey. Listen to Math & Magic on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. This is an iHeart Podcast. Guaranteed human.