the US government will talk a lot about the importance of AI reflecting American values. But what are those values exactly? Do we get to decide those values together? Who gets to decide that? Is that just Trump or is it just the heads of these companies? I'm Charlie Rossell and this is Galaxy Bring, a show where today we are going to talk about autonomous weapons in the future of AI and warfare. It's a very strange time to be covering artificial intelligence. There is a active conflict in Iran which artificial intelligence is being used. There's also this huge fallout between the AI company Anthropic and the Department of Defense over concerns about the use of their technologies. And there's this broader feeling right now over the subject of autonomous weapons that frankly these companies have this very powerful technology that they are then handing to the military. And that technology is being used in ways that maybe these companies don't feel like they have control over and that these companies are certainly worried about. There are so many moral, legal, ethical concerns. There's so much that we don't know about how these models actually work, the decisions that they make, whether they hallucinate, whether they can fail at rates different and more concerning than humans, whether humans are in the chain making these decisions, whether there are the appropriate safeguards, whether the ideologies of these companies and their leaders actually fit the ideologies of the military and whether that conflict is something that we should all be having a real conversation about. It's such a messy, scary moment and Silicon Valley is totally caught up inside of it. So I asked Will Knight to join me to talk about all of this. Will is a senior writer for Wired. He covers artificial intelligence and he writes their AI lab newsletter. Together we get into the nuts and bolts of all of this. Who the players are, what this technology can actually do and not do, and what the feature of AI warfare might look like in a moment where we are constantly escalating for fears of being left behind. Will joins me now. But first, a quick break. The world moves fast. Your work day, even faster, pitching products, drafting reports, analyzing data. Microsoft 365 Copilot is your AI assistant for work built into Word, Excel, PowerPoint, and other Microsoft 365 apps you use, helping you quickly write, analyze, create, and summarize. So you can cut through clutter and clear a path to your best work. Learn more at Microsoft.com slash N365 Copilot. Will, welcome to Galaxy, Bryn. Thanks for having me. So I want to start and I want to ground this conversation in a little bit of history, right? Because there's this long and somewhat amazing history of military innovation, right? The military building weird or niche things that then diffuse into a broader culture. Everything from radar to wristwatches to GPS to duct tape to the internet, right? So it is not uncommon, broadly speaking, for the government to fund ambitious, potentially vague technologies for the battlefield that then have all these other applications. And it's also not uncommon for them to partner with outside companies to do this work. And so I would love to go back to, you know, if you want to go even further than this, we can, but maybe starting at Project Maven and the announcement of that in 2017. The original mission is, you know, use computer vision algorithms, right? To analyze, drone footage, detect objects, and quote was turned data into quote actionable intelligence and insights at speed. What's the backstory of that? Yeah, I mean, I think that the backstory goes much further back. The sort of architect of a lot of the US thinking on AI was Dash Carter. And this was the sort of pre-chat GBT era, but in the sort of deep learning era, it seemed like, and you can see why it would fit very well with military, you know, things like targeting, like these image algorithms that could spot things. It seemed like that was going to, you know, be a paradigm shift and a really, really big one. At the time, it was incredibly controversial, right? And it's amazing how much things have changed, but you had big protests at Google which had won this contract. I think, you know, things have shifted. I think in some ways that makes a lot of sense. To me, I think the idea that you're not going to use something like AI in the world of defense seems kind of absurd, right? It's like saying you're not going to use software. I think you have to look at it too, right? Going back and looking into the Project Maven backlash initially and the fear at places like Google from some of these engineers and folks at the time was, that all sounds fine potentially, but I'm worried we're going to end up making autonomous weapons, right? This, we know sort of where the path could lead here. I think what's really fascinating about those initial protests, right, is it was over this idea of where this all leads and it seems like they're correct. This is where it is leading to, right? It's not just these vision algorithms. I would say there's always been a very longstanding principle about how you sense and make decisions and that being really critical in military conflict. You can see how the use of a vision algorithm would be very important in that. You can sort of see this trajectory towards more and more automating more of that process. But one thing I would say is having many times interviewed people at the Pentagon and people in the armed forces, there are very good reasons why people on the ground do not want to hand that off and commanders don't. As much as I think the public thinks those systems are unreliable and the idea of handing off decisions about taking other people's lives, you know, making terrible accidents is not taken lightly. And I think there is a very good reason why it is not the case that that is going to be rushed in. For a long time we've had systems that are, by many people's definitions, fully autonomous. So you have systems that will fully autonomously destroy a missile that's trying to hit a ship. There are these systems that will shoot those down and they'll have shut it down before the person can react. We do also have guided missiles that will go into an area. And so you would have a set area, you know, you can configure parameters where you know there's only going to be enemy combatants. Those are extremely expensive, the rare, what they call exquisite systems. One of the things we're seeing with Ukraine is like, you know, off the shelf drones, weaponized, and the way software can control those means that it becomes a lot easier for autonomy to be deployed. One of the things that people know is going to be really sort of game changing is swarms of them, like lots of them working together, because it's much harder to counteract 20 of them coming in to try and attack your tank. And then you can't have 20 people operating their own drones and defending it. So you're going to fight, have situations where autonomy becomes slightly more, it's going to become harder to prevent it. I think in some situations beyond those sort of purely defensive ones. I want to drill down a little bit in how these warfare models work, specifically to the extent that you know that you can help us understand, because I think demystifying all of this is very important down to the, you know, explain it like I'm five version. But to the extent that you know, like, how do they work? How many humans are there in the chains of commands? You know, how are these things, the nuts and bolts of them working? How they identify targets, safeguards, all of that? Yeah, these are classified systems. So we don't have, I don't have 100% visibility on it. The way I understand it is, it's slightly less that they're, you know, feeding in a ton of information saying, what should we do? You would see a map, and you would maybe have assets on it. And you could ask the language model to provide, ask questions about the depth that, you know, maybe the signals intelligence that was related to that particular area. And I think, you know, you would have all these sort of different resources where you could ask things about it. So I think, I think it is a little more sort of, you know, being kept to arm's length in terms of the model making, making calls. I think that they're not crazy and they're not stupid about how they ought to maybe not rely on the system at a high level. But if a language model were making some errors, and maybe the user didn't check carefully, could that lead further along to erroneous decisions? So it raises questions about the how people are trained to use those, how much they rely on this, what sort of trust it kind of maybe inspires in people they shouldn't have. It does feel like that there's probably somewhat of an analogy, if I'm gathering this correctly, between the idea of, we're journalists, right? We say we get this series of spreadsheets or this, you know, this list of data, and we want to, you know, use a language model. We let it look at this corpus of information, and it provides insights. It's not saying, this is your story, write this, it's saying, no, no, what I identify here are some patterns that we have seen. That is more of what we're talking about here when we're talking about that intelligence. Yeah, that's my understanding. And yeah, you could compare it to, you know, maybe the law or something like that where clerks are using language models to analyze a lot more case studies or maybe medicine. You might not be able to answer this question given your ability to see the systems or know this, but I feel like it's worth asking because so many people are using these generative AI tools now. So when you hear something like the Pentagon or the Department of Defense is using Claude, right? And it puts this thing into your head of like, you know, a lot of people are using these tools because they're like a friendly way to automate busy work, right? And they talk to you in this friendly way. And there's this kind of, it can feel a little bit dizzying to think about it, right? And so I'm curious from you, do you have an idea of how different this AI would be from the commercial versions of this generative AI? We do know that they gave the DOD a specialized version called Claude Gove. This is a great question. I was wondering about this, because if you prompt Claude anything to do with weapons or, you know, anything that might seem like a related to military conflict, it will say, I'm sorry, I can't help you with that. They've given it a specialized version, which I think, and they have, they've not disclosed this, but I think would have fewer of those guardrails would have to. But that does, you know, it's an interesting question, I think, is like, what does it mean when you unalign it? And I've played with some unaligned models, and they can behave in sort of surprising ways. They're able to do more things, but they'll sometimes will push back, even though, you know, that you've some of that, it depends how it's been removed, how those sort of restrictions have been removed, but you kind of have to give a model some guardrails to make it coherent. I kind of would be very curious, like what the behavior of that model is, does it sometimes actually reject stuff? Like that could have happened, especially in the early stages, and maybe that, pure speculation, but there may be that, you know, has led to some of these kind of concerns about, you know, woke models and companies or something like that. Right. And all of them, you know, there's a lot of debate between people who are real believers in this technology, and those who are more skeptical of it about personalities, right? And whether these things actually have personalities, or whether they're, you know, there's just kind of like an emergent set of traits or ways they talk. But I think that that is fundamentally extremely interesting when you get down to this level of things, right? Like, Claude is known for being a little more, compared to, you know, all the other front-year models, like it's a little more artistic, you know, it's a little more like it has its own things. Yeah, it's a vibe. That becomes very interesting when you think about the ways in which these models are being ported into this moment where, again, they're maybe not making decisions, but they are talking to a person or rather giving an input in a very specific stylized human sounding way. There are really interesting questions about that. I think the truth is most models have quite similar alignment. And I think most of those things are kind of fairly universal. There's, you know, and I think the question to me is, yeah, like, how does that really sort of change when you put in a military setting? If they've sort of buttoned it down very much and it's just doing some summarizing text, like, here's a canosica bunch of questions, that's one thing. But if it's trying to sort of do this sort of parasocial, you know, stuff that chatbots do normally, that is kind of strange. And that does really affect the user experience, the user expectations of you would probably have much higher trust in a system like that. If it were pretending to be a person, maybe when you shouldn't, you shouldn't have. We've got at this a little bit, but I think it's just worth drilling down even more on it. And break down what we mean by, like, autonomous weapons, right? Because that is a terrifying sounding phrase. And in the point of demystifying all of this, you know, it's not always clear cut. Do you think when we talk about autonomous weapons, it's best for people to think of it almost just like they have the ability to make split second decisions in anything, and that's sort of the place where we should work from? Or do you think there's actually more specificity there that autonomous weapons are actually as well, the thing that we're all worried about, right? Which are weapons that are making broader choices about target acquisition or engagement in some kind of way. The DOD has been clear that they want to always, for the moment, keep somebody making that final decision to some degree, right? That's the thing. So you can have things be autonomous, but you have a person make that decision. But they issued a directive in 2023 on autonomous weapons systems, and that spelled out that there's no restrictions on developing them. I've looked at autonomy in places like self-driving cars, in robots, and in the AI industry, like agents, right? I have not come across anywhere that is more careful about what they do than the DOD in spirit. And that doesn't mean they're going to get everything right. But I think that I wouldn't want to, I think it would be a mistake to sort of portray the people in that, in the armed forces, as wanting to do sort of autonomy for the sake of it. And I think that if they could keep their finger on the trigger, they would want to do that 100%. I think that that's, I mean, that's, it's very important context. And it's also very important to, in the sense of, there are so many different people in this chain from the people at the very top who are giving, you know, press conferences and who may be more ideological down to the people who are nuts and bolts pulling triggers. But you're making a great point when you talk about the leaders of countries, including our own, that will bear a lot of responsibility for how quickly these things get kind of rushed in. And, you know, if you see, I mean, this is one of the things with, with adoption of AI is that it's seen especially in the US as a way to kind of regain parity and overtake China because the US is, by some dimensions, you know, has the biggest military, but by some dimensions is as a disadvantage compared to China. And so leading in AI, like this is one of the real reasons why AI is such an important thing for the US government. And I think there are, there are a few questions like how, how aggressively you want to rush that in, how much you want to throw safeguards out the window. All of this comes down to when we're talking about this, it leads us to the DOD versus anthropic situation that has played out over the past couple of weeks. Because I do think it speaks a little to all of this as well, right? Which is the understanding that, that this technology, the people making this technology in some senses have strong feelings about how it should be used. Right. And what they do not feel comfortable with the US being. And then you also have a current administration that feels very strongly that they, speaking of autonomy, must have full autonomy over, over how things should be deployed. Can you just walk through the particulars of that fight for someone who's only very, you know, basically paid attention to it? Okay. Yeah. Well, I mean, there are some elements we don't, we don't have the full details of. But broadly, few matter weeks ago, the Pentagon wanted to change the contracts that it signed last year with all the big AI companies. What was in there previously that anthropic didn't want to change was specific prohibitions against using it for mass surveillance of US citizens and for autonomous weapons. So the Pentagon says, we don't want to do that. So let us change that. And in a way, anthropic just decided that was a hill they're going to die on. And I think they, you know, a lot of people in the AI world were very conscious of safety and conscious of the, the sort of moral questions around it. And so open AI stepped in, so they're going to take this contract, but they want to have some safeguards. So, but the question is, what are those safeguards? And the question is, what situations is it acceptable have something that might fail X out of a thousand times? Like, we don't know what, how much that failure is, and what times, isn't it? And I think, you know, if you were going to, was going to make the difference between telling somebody, giving some of the information that they might make a lethal decision, I think this is the question the public should have is like, where is that line being drawn? Right. Especially as this gets deployed more widely, I think it's been sort of quite tentatively implemented so far. And as that gets more capable, and these, these areas can sneak in in surprising ways, like, where is that, like, where should we and where shouldn't we? And that's kind of, yeah, that's sort of the question that's not been answered. You know, one thing that we didn't say was that Secretary of Defense Pete Hakeseth and the administration are designating anthropic as a result of some of this as a supply chain risk, which is a lot of people have said in extreme overstep of the bounds. So there's that element, which, which seems, to my sense, very irrational. And then there's the, the understanding of the actual nuts and bolts of the contract, right? Which is, okay, maybe, maybe this doesn't make sense for us. Yeah, he had this moment where the, he realized that the model, you had to get permission to do all these different sort of things. So it does sound like either the model was not as unaligned as they might have wanted, or the safeguards, the sort of checks he had to go through, the way the system was designed wasn't working for them. But yeah, that, that reaction, like, to put that in context, that's only ever been applied to Chinese companies like Huawei, who are accused of operating for the Chinese government to take that action, not just on a US company, but on one of the most important, you know, by their own measures, US companies, that there is working in this incredibly important technology seems amazingly destructive to me. And it is a very, very extreme action, but that's sort of, yeah, I guess that that's just how they roll. So something that's really important here, to my knowledge, and it speaks to this whole anthropic dispute, it speaks to almost everything we're talking about when it comes to the companies interfacing with the government, right? They create these powerful models with these capabilities that they understand will be used in warfighting. And they try to set their own boundaries to some degrees, guardrails, safeguards, whatever it is, then they have to hand it over, right? And it becomes a bit of a black box. There is this lack of understanding in specifics of how this stuff is being used on the ground because some of these operations are extremely classified, some of these uses are extremely classified. As all this anthropic fallout is happening in the news, the United States goes to war with Iran, there are bombs dropping. And so there's this moment a week or so ago, there's these first reports coming out in the press that perhaps the United States was responsible for this deadly Tomahawk missile strike on Iranian elementary school. There was all this speculation on X in the media, speculation is what it is, that oh, we're hearing about these systems like Claude being used for targeting potentially. And oh, here is a potential targeting failure. Now again, making no link or connection, this is all just pure speculation, but it speaks to this idea of would a company like Anthropoc even know if their technology was being used, or any company, let's not even use Anthropoc, would any AI company know in a situation similar to that, let's not use that one, if we're we involved? And I think that that's what I mean by the black box, which is you kind of have to give it over and I doubt that they're getting that visibility, right? Yeah, like the company that made the missile that they fired, didn't know anything about it, didn't have any say after that was used. I think as far as I understand it is not the case that Claude would have suggested that as a target. That's my understanding and I think that's right. I don't know how much they rely on Anthropoc or these other companies to help build out new applications, new uses of those models. Maybe partly Anthropoc wants to have more visibility on that because they believe they know how to do it more reliably. That would be my suspicion and I think it would sort of behoove the DoD to be aware of that as well. These people understand how these systems can fail, so you'd want to try and work with them. Remember right now, what we're seeing in the AI world that's taking off like crazy is a next generation of these tools and these are agentic AI and I know that's a buzzword, but they genuinely are and they're, you know, you'll ask a model or ask one of these tools like OpenClaude to do something, it will write code and it will go through a bunch of steps and that does raise more risk of unintended consequences because those systems are doing things and because they're going through a bunch of steps. AI is OpenClaude, it's amazing and it is really capable and you can see the allure and why it's the direction things are going to go, but it will do things that I didn't really want it to do, which is try and get something done and you'll, so how you build those is going to be an incredibly important question I think. I was preparing to talk to you and I was having a conversation about this black box potential element, right? And I said to this person, you know, they just have to hand it over and that's got to be terrifying and the person I was speaking with pushed back and said, well, is it terrifying? And their rationale was, you know, in past defense contractors, they're probably not terrified to hand over the missile or the hardware or whatever it is or, you know, the database infrastructure system to the United States military, they're probably like, they know what, where their purpose is in this chain. I was thinking about this and how the fundamental difference to me feels exactly what you're saying. It's the fundamental difference of artificial intelligence because a company like Anthropic in this situation was deeply concerned in this way that others may not have been in the past because it's not a technology that does one discrete thing. There's just an unpredictability here becoming more and more unpredictable with each iteration of this type of technology and the intelligence gives its whole purpose is to give a world of options without it just, you know, a complete and specific control. And what do you make of that idea? I think what we're likely to see in this and other critical situations is the development of new kind of approaches to engineering, actually, and how reliable those will be will be an interesting thing to watch. So a good analogy is self-driving cars. Those systems also use like non-deterministic technology. So like a vision system that you can't predict actually what it's going to, the decision it's going to make or the output, what it classifies something as. And it has to do all these things on the fly where, you know, it's like potentially life or death situations. So those companies have had to design conventional engineering around that to make sure they're safeguards in all these different steps. And that's taken years and those cars can still only go, I mean, it's amazing and they've made great progress at only going limited situations. So I think it's doable. You can do the sort of safety critical engineering with these kinds of more non-deterministic, more unpredictable technologies. The challenge is when that technology is advancing so quickly, how do you wrestle with that? Like when it's going so fast, it's hard to implement those. One of the other differences with Anthropic is where it's just not a missile-making company. People signed up because they wanted to make the world safer with the high-rise. You've reported on concerns from people in the industry, but also like the outsiders, the watchdogs, the people who are concerned about AI safety, but also just the safety in general of autonomous weapons. Can you run me through a list of some of the practical concerns that they have? There are some who, I think, just believe in a moral standpoint that handing off, we should draw a line and say, we can't hand off, so like use of chemical weapons or something. They're just worried that there'll be this sort of huge expansion of autonomous systems and that's just morally wrong. And that's a position that I think is a reasonable one to take and can be debated. And the more practical issues that people would have are absolutely like just the incredible unreliability of systems, especially if you think of a self-driving car that maybe works on the streets of San Francisco, where it's been trained for years and years, right now, but you're taking it to a kind of unfamiliar battlefield. The probability of mistakes go up. The idea that these systems would misidentify a non-combatant and take their life is a huge concern. And I think there are people on the very technical side who would just, I think, worry about the, as you were saying, you know, the reliability of systems that incorporate these more inherently unpredictable technologies. They might seem like, okay, this makes a mistake like once every 1000th times, but they can kind of compound and cause problems in unpredictable ways. We're talking a bit here about the idea of fail rates, about the idea of these models pulling in all this different information and making these insights that either hallucinations or having some kind of failure inside them. But humans too very clearly fail all the time. And so what is it about the AI of it all and these models that makes this a scarier proposition to certain people? Yeah, I think what we should be worried about, you know, so you're right, people make mistakes. And the way the military is set up is to kind of cope with that in many ways, right, to try and have this chain of command and so on that kind of minimizes the risk of that. But the thing that we should be concerned about is that these models one pretend to be very human, and they really seem human, and then they will just, they will fail in totally spectacular, unexpected ways. So it's like you're talking to an infantryman and a soldier who gives you the right answer again and again, and you're like, this guy's so good. And then all of a sudden, he's like, completely batshit crazy answer. And so that kind of unexpected thing is, is I think what is most worrying, you know, I think that's what the AI companies know, that is one, it can fail in sort of really unexpected ways sometimes, and then the people come to come to get really fooled by how how human it seems. How much of this do you think is inextricable to like the concern broadly to what has developed as a pretty reactionary culture recently in Silicon Valley? You have people like Palantir CEO Alex Karp, he was on CNBC last week, and he said, quote, this technology disrupts humanity is trained largely democratic voters, and makes their economic power less and increases the economic power of vocationally trained working class, often male voters. These disruptions are going to disrupt every aspect of our society. Now that's not about autonomous weapons or AI technology, specifically, but it is about instilling or having an ideological value inside of the company and a broader mission statement. Do you think that that is part of the reason why people are having such a strong reaction? Does that change the valence of these conversations and fears? Yeah, I don't think it has. As yeah, we started to see, you know, people sign open letters and quite prominent people, some of the other AI companies backing up anthropic in public. We haven't seen that much of a discussion of where the the line should be drawn, whether it's in the political world or that much sort of public debate or discussion about it. And it's going to be, you've got just a few people who are making really, really important decisions, whether it's the DOD or it's the CEOs. And so I would not be surprised if we saw much more pushback on it in many ways, whether it's taking jobs, whether it's being used in the military. I mean, I think we don't have a great picture of what's going on in employment in the labor market, whatever the CEO of Palantir might say. But I think if it does start to do that, people's thought to ask why these companies have honestly built tools by slurping up copyrighted material, why they own that, why they sort of in charge of disrupting everybody's jobs. And I don't know that that's the right position, but I think we could see much more of that. It seems inevitable to me too, because as we said, this isn't an Excel spreadsheet. This isn't just some, it's dynamic in this sense. It has to be instilled, whether it's anthropic talking about Claude having a constitution. In some senses, some of these companies talk about it as having values. Elon Musk has XAI, he talks about it having anti-woke values. And so then when you port this over into, yes, these systems are going to be in the chain, a chain that has decisions that ultimately can lead to people being killed, or geopolitically significant events of war, they are not doing that thing of handing over a missile that is inert and saying, use this as you will. These are dynamic systems. I think that constitution, personality, morality of those models is a whole other thing that this does raise. And it does, it is really interesting. The US government will talk a lot about the importance of AI reflecting American values. But what are those values exactly? Do we get to decide those values together? Who gets to decide that? Is that just Trump, or is it just the heads of these companies? We're on a trajectory, we're at the beginning of use of AI, and it's not going to be just summarizing things very plainly, probably for a long time. And so, whose values do they reflect? What are you trying to put into those? I mean, I think it's somewhat encouraging that if you look at any of the models, you'll take a Chinese model or a US model, most of what they'll say you shouldn't do are quite similar. There's a great Bloomberg story by the reporter Katrina Manson who notes in this piece about Project Maven about a lot of this AI warfare technology, she writes, quote, there's a palpable sense within the Pentagon that things aren't moving fast enough. Despite the show of force in Iran, officials worry that the US is at risk of falling behind. And officials are already looking past the Middle East to a potentially bigger conflict. As one person familiar with the US operations puts it, quote, Iran is an amazing precursor to what could happen with China over Taiwan. Now, that's not your reporting, so I'm not going to ask you to like talk about the, you know, veracity of those types of claims. But in your mind, what comes next year? This is obviously the beginning of something. Yeah. So just to say that that is the tenor of a lot of conversations around the Pentagon and Washington. On the one hand, you can say, well, that must reflect that this is where we're heading. And there's, there's a lot of scholars who believe that, you know, some kind of tech conflict is inevitable. There's also a sense in which those things can become self-fulfilling and arming yourself to the teeth of the technology historically has contributed to conflicts, you know, like the first World War, for example. And so what comes next, I think is sort of up to us and China, right? Just assuming that's going to happen and sort of trying to arm yourself and prepare for that might, might cause you to think that's the only solution, which I think is, would be a devastating and terrible one. And I would hope that there's different, a different path that is, is going to lead to, not lead to that. Will, thank you so much for coming on the podcast. Help demystifying some of this stuff. It's important work. Thank you for your reporting on all of it. Thanks for having me. Yeah, it was very fun. That is it for us here. Thank you again to my guest, Will Knight. If you liked what you saw here, new episodes of Galaxy Brain drop every Friday. You can subscribe on the Atlantic's YouTube channel or on Apple or Spotify or wherever it is that you get your podcasts. And if you want to support this work and the work of my fellow co-workers, you can subscribe to the publication at theatlantic.com slash listener. That's theatlantic.com slash listener. Thanks so much. And I'll see you on the internet. This episode of Galaxy Brain was produced by Renee Clark and engineered by Dave Grine. Our theme is by Rob Smersiak. Claudina Bade is the executive producer of Atlantic Audio and Andrea Valdez is our managing editor.
