Hello, everyone. This is Tom Uren. I'm here with another Between Two Nerds episode with the gruck. G'day, mate. How are you? Fine. And yourself? I'm good. This week's episode is brought to you by Airlock Digital, who make whitelisting easy to carry out in your organization. Particularly important now that AI-powered hackers are running around everywhere. Speaking of which, so this week... That was a smooth segue that you set up right there. Speaking of which, this week there was news that Hugging Face was hacked. That is an American AI-adjacent company. It's heavily involved in hosting models and is quite central. Research and stuff. And the story is that an open AI model that the company was testing, open AI was testing, on a cybersecurity evaluation, It had had its safeguards backed off. It, the OpenAI model, hacked its own company infrastructure, got out to the internet, and then hacked Hugging Face in an attempt to find the answers to the cybersecurity evaluation. So that's... That's a passing grade in my book. Yes, you would think so, wouldn't you? Congratulations, yeah. So in the wake of that incident, you messaged me and said, I'm just thinking that cyber is people. Right. And so we're going to talk about what that means and whether AI changes that. My first thought was just cyber power and states. But anyway, you kick off. Right. So my argument for cyber is people. There's a lot of stuff where people like Western aid to Ukraine helped with cybersecurity. And I'm looking at it and it's like it helped with licenses. It helped with deferring the costs of cloud computing. It helped with that sort of stuff. But all of the actual work had to be done by human beings. And those human beings were Ukrainian administrators. So the Western aid was basically like paying for the Uber ride. But the guy who drove the Uber doesn't get any credit for like taking you somewhere, which is I mean, that's maybe fair, but someone had to do it. Right. Someone had to do the work. There was like all of the cyber stuff that you can talk about. None of it is actually computers. It's all actually people. If you've got a firewall that that firewall has to be configured, it has to be installed, it has to be monitored like that has to be done by people. So the firewall is a tool being used by people. So the cyber stuff is just like it's people doing things, like all of it. And so that's where I was coming from. I was just very frustrated at the idea of like, you know, Nmap by itself is not a hacking tool. It's just a tool that does a thing. If you use it for hacking, then it is. But that's the person doing it. Like objects don't have agency. so in the case hugging face case the apparent story is that open ai set the model there was seems like a couple of top tier models in front of a computer uh metaphorically and said we want you to go hack this off you go and they did everything that i described so the breaking out of open ai's infrastructure the hacking of hugging face autonomously so there's no actual person i mean there's a person to say go and then interestingly the hugging face crew they said that they used another model an open weight chinese model to do instant response because just the volume it would have taken them quite a long time and they said that they were able to rip through the incident in a couple of hours using that model. And so in this case, there's an attack, supposedly autonomous, although directed by a person, set off by a person, and an incident response, again, run by a machine, but directed by a human. So I'd say that one of the other parts of Cyber as People is OpenAI has a network that was set up by people. and that network was set up in a way that the test environment was not completely isolated just a bit and that's a mistake perhaps yeah and that's a mistake and one of the things is people make errors like humans make human errors so like cyber as people is the like all of the attack stuff is people doing things but on the defense it's people doing things as well like they make mistakes or they cut corners or they misconfigure something or they, you know, get lazy and stop doing what they, you know, they're supposed to do. They go from logging in with a password to leaving it logged in all the time because it's annoying to do otherwise. Cyber is people. And so I would say that even though in this case, there's a lot of AI involved, there were still people that allowed the cyber to happen, that like created the environment in which that was possible so it's still in a way it's people i don't know that that's always going to be true right like it feels like it might be the end of that but right so yeah when you sent that to me i had a little bit of think about and i read it as cyber power is people and certainly uh so not very good at reading but certainly or maybe no you're not very good at remembering what you're saying but certainly like three years ago if you had said cyber power is people in a state context i would have said absolutely yes so i think the reason that we've got some countries that we would consider cyber power powers is that they just have the political will to harness a lot of people to go and do cyber operations so yeah like north korea for example who don't have other things that like you wouldn't look at north korea where like the lights are not on at night except in like the capital city like you wouldn't look at that and be like this is a cyber powerhouse yeah this is where you want to go for like top tier talent yes look at and be like these are subsistence farmers living in the stone age essentially yeah i think that like but that's a yeah there was a long period where cyber security people were skeptical of north korea for i think exactly that reason right right right and then And now you'll see like multiple change bugs to get access to a thing where they change a JavaScript file for 15 minutes in the right window to capture something. And like they transfer several million dollars from a hardware wallet that wasn't accessible except at that brief window. Yeah, so. You know, like they're doing magic level, state level, you know, state level, state tier level at least. Like there's higher level stuff, but they're definitely very capable. I was to argue that cyber power remains people regardless of AI. I mean there is a lot of evidence that North Korea is using AI for things like its IT worker schemes where it uses it to create resumes where North Koreans can get jobs in Western companies and earn an income and perhaps hack them right So that is useful But if you I still believe that if you took all the AI away from North Korea they would still be pulling off outrageous hacks right Right. Maybe not as quickly or as many, but still. I think it would be the volume that goes down. That's all, right? AI doesn't give them a capability that they lack. It just allows them to scale better. That's the way I would interpret it, is that if you want to have cyber, you have to invest in cyber, even if that turns out to be an AI. It's still an expression of state will to have state power. And if you don't have the will, then it doesn't matter what tooling exists, you're still not going to have, like it's not going to be magically available just because AI can do it. Yes. So another thought or question I had was, does the rise of very powerful AI change the cyber power rankings power rankings and where you know maybe maybe you would face the us or china on top you could argue about that yeah but again i don't know that that makes a huge difference i think that they will each have tools that are quite good for doing things and yeah no i think as you were saying earlier just before we started recording it's that the countries that are investing the sort of resources to have like 1,000 hackers right now. Like those are the same people that are investing the resources to have a lot of compute and a lot of like to develop frontier models and all that. It's the same people, right? So it's not like suddenly the Sudanese are going to become a cyber power just because all they need is a few NVIDIA chips and like access to hugging face to download a model. We're not there yet, at least. I wonder if we will ever get there, because I feel like the counter side of that is if you can download a good hacking AI, you should be able to download a good defending AI that can configure systems for you so you don't make those mistakes. Right. So it's a rising tide lifts all boats kind of situation. I suppose you could think that maybe the US might end up a bit better because they've got the frontier labs right now but I think it's like it's not a magnitude of water better off right it may be a little bit better off on the edges on the margins I mean it's like they're the world's only superpower and that will be true in AI I think but it doesn't it doesn't necessarily translate into being a magnet as you were saying like it doesn't mean that they're like magnitudes better than other people who have to sort of follow behind on that. Yeah, yeah. Maybe the Chinese will be second because they've got their own labs and then poor old Russia will be living off the dregs of the Chinese. Chinese second-hand AIs. Now, I was also thinking about cyber scammers. So the big industrialized, it's often called pig butchering. Yeah. Yeah, where they have like compounds even. Compounds of people they've lured their own false pretenses and they kind of like romance scams, crypto investment scams. And these have become massive enterprises. Huge industries. And they seem like an example of where cyber is. People because they've become huge because they've been able to harness like tens of thousands or hundreds of thousands of people, right? And get them to work. Yes. But they also seem like a business that AI could really replace a lot of those people. It's ripe for disruption. There's an opportunity for the first agentic AI that does pick-buttering scams to go in and really shake things up. You know, I was actually going to say that, but then I thought that might be a bit insensitive. So I'm glad you did it. Thanks a lot, Tom. So I actually had really mixed feelings about that because like scamming is bad. Having enslaved people do the scamming is even worse. So, you know, AI replacing them is, is that a win? I'd probably have to say yes. It's a win for humanity, but not necessarily for victims. Yes. Well, you remove half the victims, right? Right, right. As someone who uses AI, we both know, and I'm sure our audience knows, the one thing that AI is really good at is convincing people. It's very good at being confident and just saying things that it believes are true. Right, right. Getting you to accept what it puts out there is true. You often feel like you're being dragged in different directions because I'll say this and I'll go, yes. And then I'll say, but what about this? And you'll go, you're perfectly right. Yes. And then you're sort of back where you started. And it's worth bringing up the tension. What you're saying is... There's a guy on YouTube, Mo Bitar, I think his handle is, who's very entertaining. And he speaks a lot about reinforcement learning by human feedback, where they will train the AI based on what a person likes. So there is an element of like they're deliberately trained to give you something that will be pleasing. If they just said, you're an idiot. Why are you wasting my time? You're probably not going to stay on that chat client very long. You're not going to be like, hey, I should respond to that. Let's keep this going. I guess going back to scam compounds, that seems like the exact sort of property that would be very useful, right? Right. And I would say that one of the things that holds back AI at home or wherever is that it's very slow. Because you don't have the compute that's available to these frontier labs. Instead, you have whatever you can afford, which is less money than they're spending. And so, you know, running an AI on your laptop could take an hour to process a, like, whatever. That would take, you know, less than a minute if you're using the hardware from a frontier lab. However, because email scams, like these email-based scams, are asynchronous, that cost is not a problem. Right? It's not a thing that gets in the way. If you can only do a chat message every 15 minutes instead of every one, you can deal with that. That's absolutely fine. That's not an issue at all. So I'm bearish on human trafficking and bullish on AI scanning. Yeah, because it seems like the people who they've trafficked and are running the scams aren't just winging it. They've got, I guess, empirically developed playbooks that they know work based on where you are in a conversation, what you try next. And that seems like if you've got such a strong framework, you wouldn't need a very super powerful AI to run a playbook, right? Yeah. It needs to modify a template as opposed to figure out what to do next as a step towards achieving a goal, right? It's you get in a thing that says this, you look through your playbook to see, okay, the appropriate response is, you know, 17B. You pull it out, you, you know, read Ziggard a little bit. So it's not the exact same text. Yeah. And whatever. And yeah the perspective of a scam Kim Kim it seems like using machines instead of people would be good from the perspective of like just management overhead Like I don want to have to deal with managing like the logistics around a whole lot of people but also from the perspective of getting caught and getting punished. So right right so it's if if if China finds out that you've stolen a lot of money, that's one thing. If China finds out that you've kept, you know, 10,000 of their citizens in slave conditions and been beating them and starving them, that's a death sentence. For you and everyone involved, yeah. Right. So, yeah, if I'm a scam kingpin, I'm looking at this and saying, this saves me so much. It saves me grief if I get caught. it saves me money because I don't have to maintain a compound with thousands of people and hundreds of guards and getting in food and keeping it. Like you don't have to pay for any of the stuff that you have to pay. Like you don't need dormitories. You don't need like working rooms. You don't need a thousand computers. Yeah. You need 10,000 computers, but you can keep in a warehouse. Right. Yeah. So, I mean, I buy the logic that for them, cyber power is maybe, could foreseeably be more. It's not people. Yeah, it's not people. And that makes me wonder what's wrong with my previous logic when it came to states and, like, what's the difference between a massive scam compound and a state? And a state? What? I mean, other than the obvious differences. It comes down to motivation or what are the objectives that you're trying to accomplish? And if the objectives you're trying to accomplish is trick a whole bunch of people and get them to do something or just convince a whole bunch of people that you are trustworthy and correct, AI is great at that. That is a good thing. And if you run a business that requires the ability to make people trust you and like you and do what you say, AI is very good for that. And so scammers are an excellent match. Like, say, Google or Meta or… Yeah, like, they're good matches as well, I think, at some point. But, you know, like, I would say that that is useful for states because if you're doing phishing, that's what you're trying to do, right? So, like, if you're a state cyber capability, then having better phishing that requires fewer people to manage it allows you to scale better, to do more custom-target stuff, gives you access to languages that you don't necessarily have at native level, right? So you can do recruitments against, like, someone who, you know, you could do it in their native language, right? Which would be good because if you can't write at native level, you can't convince them that you're a fellow native. So I think it's useful in this sort of case. But if you're a state and you're like so interested in Romania that you need to have someone who can write at a Romanian native level, you very likely have people like that on staff because you didn't just get interested in Romania today. Right. Right. It's sort of been an ongoing thing. And so you've been getting in experts, getting in analysts, getting in like all of this stuff. So as a scammer, being able to now speak like Malay seems useful. or like you know being able to speak Vietnamese in your scamming stuff that seems useful I don't know it's useful for a espionage unit because if you need to like do stuff against Vietnam you have Vietnamese capabilities already otherwise right you wouldn't be interested I mean I guess what I'm hearing is that you think there's a difference in the sort of depth of interest so that a state really, really wants to achieve something in particular. And so AI is helpful. They've been investing resources into it for a while already, I think. Yep. Whereas the sort of scam compound, it doesn't care who you are. Right. And so it's anything that will get us a small percentage of a lot of people is good enough. And AI is good enough. because you don't need motivated workers because obviously we're forcing them to work. I mean, I guess you don't need the top tier of worker, perhaps, is a better idea. Right. So, like, as a scammer, being able to troll and cast a wide net is very useful because you're working on percentages, right? You're going to send out however many attempts, and some of them will be successful. And some of those will lead to actually getting money. And of those, there's going to be a range of money that you get from a small amount to a very large amount. And so somehow that small number of whatever has to cover everything. So the larger net you can throw, the more money you make. And if the costs of throwing a net increase with size, which it would if you have to get more people, If you have to traffic more people to do work, you're paying more for that net. And so it's going to impact the economics. Whereas with AI, I think you get a very large net for a fixed spend. And it gives you a larger net overall. And so that seems very useful. But states are harpoon fishers, right? Like a state doesn't need to throw out a very, very wide net and just see what they get. because they don't have the resources to process that. And it's like, it's not a thing that they want to do. They want to be able to do like these exquisite, you know, important things against people that matter to them, not against, you know, every Joe Schmo out there. Right, right. So what I'm hearing is that scams, pig butchering, they have a wide net where they don't care about anything in particular and AI is good enough. It's just particularly well suited to computers where you get people to like you and it doesn't matter if you screw up all that much for any individual case. Sort of, you know, law of averages kind of stuff. Whereas in states, the jobs are much more diverse. There's places where AI really helps, also places where it's not that good. And you actually really care about getting it right because it's, you know, an intelligence priority rather than just, no. It's not a side project where you're like making a birdhouse in your garage. And if you screw up putting it together, whatever, you'll just toss it and make another one. You'll get it right eventually and that's fine. Like it's not a hobby. Yeah, yeah. You're not digging around. Yeah, that makes sense. Which I mean I feel like that almost why the open AI and hugging face instance shows why AI cybersecurity would not be good for like offensive cyber by AI is not a good spend right now anyway for a state because the last thing you want is to get close to a network that you need to be on tell your ai you know solve this problem for me get on that network it decides that it's hard to do it directly so it it veers off to the left and acts a whole bunch of random stuff in some long convoluted path to eventually get to the network And like, that's fine to do if that's what you want to do. If that's the pathway you've figured out, it's not fine to do because something autonomously decides, screw it, you know, we'll do this. And, you know, raise your risk profile and raise the opportunity of being discovered. So I guess in that example, what you're saying is that the open AI models had no concept of risk appetite and had no concept that they were no idea that they were exceeding that risk appetite. I'm sure I'm sure open AI did not want to write that press release. No one at the organization wanted that. And for OpenAI, that's perhaps, I mean, it's a hilarious outcome. But for a state when it's hacking, I don't know, Xi Jinping, that's not quite so funny. One of the things that occurred to me, like the first time I saw just the hugging face report before OpenAI had figured out it was them. When it was this very, very breathless, you know, like there were all of these, this swarm of AI agents that were doing all of this stuff. and like they did all this crazy whatever. And I was looking at it and I was like, that's really expensive. Who would pay for that? It just like, that's what didn't make sense to me at the time. It's like, who would have that much compute that they could spend on hugging face? And so the fact that it's a frontier lab with one of their training runs that got away, like that actually matches, right? It feels in some ways a great story and also totally unsurprising. And I came across another tweet where Vercel, which is a company that hosts AI models and runs them, I think, it was talking about how much it would cost to assess a code base for security vulnerabilities, which I assume is pretty much the same as it would be to do, you know, assess it for security vulnerabilities if you wanted to hack it. And they were talking about for a company. Just to pick an example at random. Yeah. Using OpenAI's top available model, it would be six figures, they said. And so for a state that seems like, oh, yeah, that's something that is totally feasible. And especially if we were going to, you know, it was some underpinning software we could amortize that over many different hacks right right so like a state saying like um xi jinping's phone is so important to us that we're willing to spend up to 50 on a chat gpt max license no wait a hundred dollars on a chat gpt max license and see what happens all right like that's just not the case right they they have they get budget that they invest to do these things. And so that sort of cost is just a cost of doing business that you would accept. A six-figure price tag is just, that's what it costs to do this, to use this tool. We've determined this is the right tool for the job. That's what we have to pay. You have to justify it and someone has to approve it, but it's not like we don't have that money. We can't do this. It's a different set of problems as opposed to just like, I don't have $100,000 to spend on auditing software. Well, I feel that that is actually the thing. If it comes down to a narrow set of tasks, there are some things that cyber is very good at. And I can envisage cyber being very good at both breaking into and defending networks because they're concrete tasks. There are yes, no answers. you can know if you've patched something and know if you've not you can you haven't right counting the number of computers you have there is a finite number of computers that you have like asset management is the thing that you can do because there's a quantifiable solution like there's a quantifiable answer of like the number of assets that you do have yeah yeah yeah no yeah so yeah absolutely now i think attacking and defending networks are very narrow jobs like Like the vast majority of humanity does not give a rat's ass about those jobs. The vast majority of humanity is wrong. Even in the field of cybersecurity, the jobs are much broader than just attack and defend narrowly. So for a whole state, when you come to expressing cyber power, there's so many different ways to express it. And there's so many different jobs. It does come down to people still. Yeah. So like in a way, this does go back to a post I wrote in 2000. Basically, I wrote in 2016 or 17, I think 17, which was like how to make an APT. All right. And right. So it was like you need hackers. Right. But then you need sys administrators and then you need a manager and then you need an admin and then you need like all this stuff. And so it turns out that your 15 people that you need at a minimum, four of them are doing hacking. And the rest of them are just doing all of the other things that you need. And so I think if you replace the hackers, then you still now have 11 people doing cyber, even though none of them are hands on keyboard anymore. And I would say that, you know, that is going to hold true because AI can sort of help automate some part of those jobs, but it can't replace the people that are involved. And I don't know that it ever will. So I think that everything we've said has kind of just reinforced what I thought at the beginning and I wasn't sure of that. yes particularly when it comes to cyber power that is people you're and i think that will be kind of enduring because right and it's because there's such a wide scope of things that a state wants to do that that you'll need people but when it comes to cyber is people like a more narrow right on the well and i think maybe you were using it in a broad sense but i'm using it in a narrow sense that I'm not convinced that AI will not change what exactly it is. And I think that's because if you've got a narrow conception of what cyber is, AI is actually getting very, very good at that. And so I guess it comes down to what you think cyber really is. Yeah. So cyber is what we make of it. And that might be people. Thanks, Greg. Thanks, Tom. Bye.
