Hey everyone, I'm James Wilson, and welcome to this week's Seriously Risky Biz. This is our cybersecurity policy and intelligence podcast, and it is based on the excellent newsletter that my colleague Tom Uren writes, Seriously Risky Business. You can find Tom's newsletter over at our website, that's risky.biz, where you can view the newsletter and subscribe to it so that it lands in your inbox every week. Tom will be along shortly to talk through the newsletter, but first, quick thanks to our sponsor for this week, Sondera. Sondera makes a sort of an AI agent harness and framework that, well, I've interviewed Josh, their co-founder and CEO recently, and he essentially explains to me that it's a way to teach your agents the rules of the road so that they don't go rogue, and it's a very interesting technology. Do check it out. But on to Tom's newsletter. So Tom has two topics for this week, and the first one is ransomware uses AI to amp up negotiations. And the second is the bugpocalypse is here officially. So two very interesting and topics that are near and dear to me. In the case of the ransomware operators using AI to amp up negotiations, Tom talks about how this is an interesting strategic pivot from groups that were previously using AI to find their exploits, to find their way in. there's now groups emerging that are actually using just simple techniques to do the actual ransomware and extortion part but then leveraging ai to actually handle the negotiations and if you think about it it's kind of brilliant because if you're a non-english speaking ransomware actor well the negotiations probably been the difficult part in the past and then we get talking about the bugpocalypse uh you know it's officially here if you look at the bug numbers but you know as tom and I talk about, it's still kind of a head scratcher as to like, okay, all these bugs, but where's the wreckage? Where's the carnage? We're waiting. We're getting bored. But nevertheless, there's some interesting implications around this, especially around, you know, this perception that, you know, patches will save us when, of course, as Tom and I discuss, that's not actually the case. But listen, I'll drop you in here where Tom starts to talk about this interesting pivot in the ransomware ecosystem where we're starting to see AI being exceptionally successfully leveraged to handle negotiations. Enjoy. Yeah, so the, I guess, conventional wisdom or the conventional fear that many people have is that AI is getting so good at cyber stuff that the logical outcome is just more hacks and therefore more ransomware. This particular ransomware group called Fulcrum Sec. They haven't been around for all that long, but what I find quite fascinating about them is that instead of using AI to hack, they get in through simple means like having creds. They use AI to basically turbocharge their negotiations to try and give them more leverage. So they've hacked a number of companies. They've taken vast amounts of data of all different sorts, So from different cloud storage systems, from GitHub, a variety. And then they actually spend the time to analyze it and try and make sense of it. So in the other podcast I do, Between Two Nerds with the Gruck, he's spoken often about how you can get a vast dump of data, but no one is going to do the work to try and pull out the gold nuggets. And so just dumping a huge amount of data is not very impactful. I don't know if they're fans, but anyway, the ransomware group, one case they said they spent six months analyzing the data they'd stolen before they went to the victim. And what it allows them to do is say, we've stolen this data. Here's exactly why it's important, at least according to the LLM. Here's proof that we have it. If you need proof, we can give you any file. and that's why you should pay us this ransom. Right. And Fulcrum Stack, as you mentioned, not a group we'd necessarily heard much from before, but this particular attack that you dived into, it's not small, right? This is the Novo Nordisk 1, makers of Wegovi and those GLP super drugs, but both a very large target and a very large haul of data as well, right? Yeah, yeah. So that one, it's interesting that they extracted a number of proprietary AI models from Novo. And so the history is they hack, they pretty quickly turn that around and say, we've analyzed this data, present Novo Nordisk with a ransom When that falls through when the negotiations fall through they then have basically a nice report wrapped up in a bow that they can hand to a journalist or a threat intel investigator And the idea is that when they republish this that adds additional pressure to the victim. So this negotiation fell through. They reached out to data breaches.net, which reports data breaches. And they basically have this wonderful report of here's all the things that we did. Novo's security was absolutely terrible. It's really surprising for such a big company. Here's the intellectual property we extracted. And they said that in this case, they took a number of AI models and they used AI to analyze the AI models. And then they said they used a team of agents. And the idea is that by laying out a case for the And that wasn't the only intellectual property. But by laying out a case for this is the value of the intellectual property, their security was terrible. My feeling there is that's a negotiating ploy to say, oh, look, here's something that would be very attractive to a class action lawyer. And so, you know, that's another reason to pay us a ransom. Now, curiously, Data Breachers reports that basically at the same time, there was another ransomware data extortion attempt from Novo. So someone else broke into Novo, they stole the data. And then the next day, they basically said to Novo Nordisk, we stole this data, we want $50 million. Like intuitively, it just seems like to me that fulcrum sex approach is a lot more compelling, because they've got reasons, not just we've got a random bunch of data. We don't even know what it is. We just know the size that we've stolen. In this case, it didn't pay either of those ransoms. So I suppose good on them. But I think it's a really interesting strategy. I didn't dive into it in the report, but Fulcrum Sec also does things like, say, we extracted a whole lot of data. Some of it's personal and sensitive, and we're not going to leak that. And in one case, they actually reached out to the victim and said, we don't know how to redact this safely. You know, redact it safely for us before we leak it. So they're very much portraying themselves as, you know, we're doing this as ethically as we possibly can. The ethical ransomware. I mean, what a wonderful development. And, you know, talking about the strategy of this, I think the thing that's interesting is this is a brilliant example of where, you know, AI is applied to the thing where, I guess, there was no other simple solution, right? to your point, they were using simple methods to hack in. This wasn't complicated zero-day exploits. It's just stolen creds. But am I right in thinking that the interesting strategic element of this is that if you're a non-English speaking ransomware group, trying to negotiate has always been quite a tricky thing. There's the language barrier. There's perhaps not understanding different jurisdictions and elements, right? And this just feels like if you're going to use AI somewhere, well, yeah, maybe this is the most brilliant point to do it. If you've got every other element of the ransomware chain solved for? It really depends what you think your rate limiting step is. What is the thing that stops you making more money? And one view of ransomware is that it's a hacking endeavor, which is, I think, the default view for a lot of people in cybersecurity, because that's their mindset. They're technical people that like technical stuff. And so hacking ransomware is a hacking problem. Another view is that it's an extortion problem. Like how do you convince someone else to give you money? And like the way to convince them to give you money is not necessarily by doing more hacking. So the broader trend in the ransomware industry is that it's shifting towards data extortion because one of the reasons is that companies are just getting better at backups. And so you can encrypt something, but if they've got a backup, then well, whoop-dee-doo, you know, that's what we've trained for. But if you've stolen their data and threatened to releak it, well, having a backup strategy doesn't help there. It's also less painful for the ransomware crew, like managing an encryption algorithm, managing the keys, making sure we've got the right keys and they can decrypt their stuff and not making mistakes, that's a lot of work. But at the same time, payment rates have also been falling. So one way to approach that is just to hack a whole lot more stuff and try and make up for lower payment rates with more volume. Another way is to try and make the most out of each hack that you get. So I just find it really interesting that this particular group has invested in the low volume but trying to squeeze as much juice out of each hack that they get Yeah it really interesting to actually get a chance to sort of examine and compare the emerging different business models for ransomware groups and see what their success will be like. And I think one of the novel elements that I find here that I think will lend itself to quite a bit of success for them is that we already know we're in a position where we've got large language models, both finding all the bugs as well as potentially generating some bugs and creating the exploits, et cetera. So it almost feels like the act of finding initial access through a zero day or whatever else is almost already approaching a zero sum game where models on both sides are doing both the attacking and defense or increasingly autonomously. Okay, so that takes that off the table. What I found interesting here was the thought experiment of how long until the ransomware negotiator on the receiving end is also a large language model. I think for the example I spoke about, Novo Nordisk, that's perfectly plausible because I don't think they were ever going to pay in the first place. It seems like they had a policy that was, we do not pay. The Fulcrum Sec, they actually told data breaches that it seemed like they were just playing for time to manage incident response. So like, why bother paying a person? An LLM can do that just as well, if not better, than a person, I think, because you're not really trying to achieve anything. You're just trying to delay. And I think, you know, you can talk to a chatbot all day and they're perfectly happy doing it. There's no angst. And it seems like the companies that are not paying are the ones we hear these stories from because then fulcrum sec will basically reach out to someone and and we see it published in different places they're quite up front about using an ai in their analysis in one case they said yeah we analyzed their data using the chat gpt account that we stole from them so it's just adding insult to injury uh it's it's beautiful work i mean it's on one hand, it's like the ethical hacker, but also, you know, yeah, we'll use their own AI account. You know, how it goes off to them. But I'd like to your summary here. Our take home message here is that nerdy, technically focused ransomware actors use AI to hack, but sophisticated ransomware actors, they use it to negotiate. But good segue, because something that now seems to be very much non-negotiable in our landscape is the bug apocalypse. And your second article here makes the bold claim that it is officially here. And so what has led you to this conclusion? Well, it was really the trend micro. They have this thing called the Zero Day Initiative and there was a write-up in there. They talked about the bug apocalypse officially arriving. It's just the sheer number of bugs that are being patched every patch whatever, whatever day, patch Tuesday for Microsoft. And so the number of bugs this month doubled over last month, which was a record month. And I think there was something like 500, 600 Microsoft-specific bugs. And then there was also another 480 in Chrome that then get plugged through to Microsoft Edge. And some report said that Google had patched over 1,000 bugs in the last month. I wasn't able to find a really good source for that because it's broken out in Android, Chrome, whatever. And so just the pace and scale is like surprising. I think the other thing that's surprising is that so far we haven't seen the internet meltdown over this. There've been the odd story of here is a, it's like a kind of shiny butterfly that attracts attention. There's hacking and there's AI, but it hasn't yet really amounted to significant impacts. Yeah. And that was my comment to you yesterday when we were chatting in Slack last night was that, you know, I almost pushed back a little bit on this notion that the bug apocalypse is here officially because it's like, well, it might be here, but its impact is still on its way, not going to happen. Is it apocalypse or is it just bugs? Yeah, exactly. Yeah, I did go back and forth about the title. And the Grek and I actually spoke about this in a Between Two Nerds discussion a couple of weeks ago, I think. The impact, essentially that there's lots of bugs, doesn't seem to be much impact. One of the things I dive into this piece is the entire industry and I think policymakers are saying, assuming that the right answer to this is to to just patch like crazy and keep patching. And keep patching. And I mean what happens if the number of bugs doubles next month and then the month after that and then the month after that like is it a never bug hunt um and it so there is some academic debate whether there's a finite number of bugs or infinite and so i think it's interesting to think about what happens if this strategy of trying to patch our way is just misguided yeah But the main point I make in the piece is that, in fact, patching solves problems for some organizations. So if you manage AWS or Azure, you can patch your entire cloud. That is definitely a win, removing a vulnerability from everyone who uses it, a plus. But for on-premise software, there's a lot of people just never patch. yeah or can't patch or don't even know they've got something they need to pass exactly yeah yeah they're just incapable of doing it so if you look at each year you know the different reports that look at how breaches occur unpatch vulnerabilities is a significant factor every single year yeah and so ironically by issuing all these patches we're actually creating more and more unpatched vulnerabilities because we know that many people will never patch. And so, you know, the winning strategy is to actually create more vulnerability. Which I think is a, I can't see, I can't see that it's wrong, but it also seems like what's the alternative? I think we have to patch. Well, that's the thing, right? I mean, I've had, you know, counter arguments from Brad Arkin, who's very clearly made the point to me that, look, patching is not the way to safety. You've got to think about what are these bugs teaching you and do those mitigations. And I get that. But then, you know, talking to Carsten Knoll, who's a security researcher, where his stance was, well, if you know the bugs there, why wouldn't you patch it? And so I think that's kind of the confluence of these two things, which is there is no defensible argument here that says we don't patch and that we don't fix these bugs. And so we do inevitably end up with these large volumes of patches. But I think the, you know, in essence, what I took away from this piece that you wrote is patching itself is not all good news, right? There's the point you just talked about around how not everyone patches. Sometimes they can't. But the other flip side to this is a patch is actually a signal to an attacker of, one, there is a vulnerability, exactly which versions it is. And now with the advances in AI, it is trivial to turn a patch into an exploit because essentially the fault that it can exploit is, or the bug that it can exploit is right there. Yeah, so it's a problem. I think one way to think about it is the avalanche of patches is actually a leading indicator for chaos to come for those reasons. It's easy to reverse engineer patches to get to exploits. and at some point that's what some people will start to do. And, I mean, it's only been a couple of months of outrageous patching volumes, so plenty of time, plenty of time. Plenty of time. But also, you know, going back to that point of, you know, where's the wreckage, in the last sort of week or two I've been doing my own little side project of security vulnerability research and I will say one thing I've observed is that a decent large language model and a harnessed will constantly shake out bugs, even out of things that should be really well hardened. Like I've been throwing it at parts of macOS because that's what I'm familiar with. And I found really interesting stuff that surprised me. But the amount of interesting stuff that I can turn into an actual exploit chain that's worth even reporting, after a couple of weeks of work, that has amounted to only one submission. And so I think people need to understand that those, let's take the example of the 570 bugs in Microsoft's Pats Tuesday, those could well be 500 actual isolated code quality memory corruption issues that are valid bugs. But reaching that bug from the network, reaching that bug from a logged-in session, there's still a world of work that has to be done here. And so I kind of wonder if that's the reason there's this delay where it's just like there's patching all the individual bugs, yes, but soon I think we'll start to see enough of these have been patched and identified and given signals to attackers that they're then able to string them together into these interesting exploit chains and maybe then we start to see the real wreckage It's a brave new world, James It is a brave new world, Tom Well, mate, let's wrap it up there and again, wonderful newsletter this week thoroughly enjoyed reading it and look forward to chatting with you next week Thanks, James you
