Risky Business

Risky Business #837 -- GitHub Actions footgun claims TanStack

65 min
May 13, 202618 days ago
Listen to Episode
Summary

Episode 837 covers a critical supply chain attack on TanStack via a misconfigured GitHub Action that deployed an NPM worm, infrastructure vulnerabilities in Instructure Canvas and Palo Alto products, and emerging threats from AI-powered vulnerability discovery and deepfake technology. The hosts discuss how GitHub Actions present dangerous attack surfaces, the prevalence of AI in both offensive and defensive security, and the importance of offline resilience for critical infrastructure.

Insights
  • GitHub Actions on pull request triggers represent a critical footgun in modern CI/CD pipelines—the advisory exists but developers still misconfigure them, enabling supply chain attacks at scale
  • AI-powered vulnerability discovery is now at industrial scale; traditional memory corruption bugs in security appliances are being found and exploited faster than vendors can patch them
  • Deepfake detection technology is failing at scale—state-of-the-art detection models misclassified nearly 100% of real deepfake samples as authentic, indicating a fundamental gap in defensive capabilities
  • Organizations face a paradox: building offline resilience for critical infrastructure introduces new attack surface through redundancy and complexity, yet avoiding it leaves them vulnerable to targeted infrastructure attacks
  • LLM-based security products face unrealistic perfection expectations compared to traditional ML solutions; customers expect zero mistakes despite the technology being fundamentally probabilistic
Trends
AI-driven vulnerability discovery becoming standard offensive capability; organizations must assume attackers have access to automated vuln-finding toolsSupply chain attacks shifting from credential theft to infrastructure misconfiguration exploitation; trust boundaries in CI/CD pipelines are critical new attack surfaceDeepfake technology commoditization enabling real-time video impersonation attacks; identity verification via video is becoming unreliableCritical infrastructure operators forced to plan for internet disconnection scenarios due to geopolitical tensions and demonstrated targeting of cloud providersHype cycle repeating: AI in security following same adoption pattern as machine learning 10 years ago, with vendors bolting on AI features for funding/purchasing authority rather than solving specific problemsRansomware-as-obfuscation tactic emerging; threat actors using ransomware to muddy attribution and slow victim response, not necessarily for ransom paymentSatellite internet becoming strategic infrastructure; Russia, EU, and China all developing independent satellite networks to reduce dependency on US-controlled systemsMulti-user Linux security assumptions finally being challenged; local privilege escalation bugs (Dirty Frag, Copyfail) demonstrate kernel cache poisoning remains viable attack vectorDNSSEC validation failures showing that perfect security mechanisms fail in practice; pragmatic degradation (disabling validation) preferred over strict enforcementAgentic AI in security products requiring new evaluation frameworks; customers asking about red teaming, training data, autonomy levels, and explainability before deployment
Companies
TanStack
React ecosystem library compromised via malicious GitHub Actions pull request, leading to NPM worm distribution
GitHub
Pull request trigger GitHub Actions feature enabled supply chain attack; company issued advisories but misconfigurati...
Instructure
Canvas learning platform breached by Shiny Hunters; billions of student-educator messages stolen; ransom reportedly paid
Palo Alto Networks
Memory corruption vulnerability in content length header parsing; remote code execution in captive portal component
Avanti
Endpoint manager plagued by repeated vulnerabilities; 30-year-old legacy codebase with ongoing security issues
Cloudflare
Disabled DNSSEC validation for entire .de TLD after key rotation failure; pragmatic decision to maintain availability
Google
Threat Intelligence Group released report on industrial-scale AI usage by adversaries; discovered threat actors using...
Sublime Security
Modern AI-enabled email security platform; sponsor interview discussing customer evaluation of agentic AI features
Foxconn
North American factories experienced cyberattack with limited impact; systems restored quickly
SpaceX
Starlink satellite internet system; Russia developing competing system with different orbital characteristics
Shiny Hunters
Ransomware gang responsible for Instructure Canvas breach; reportedly received ransom payment
Muddy Water
Iranian APT group using Chaos ransomware to obfuscate attribution and muddy investigation waters
Karakurt
Russian ransomware gang linked to Latvian hacker Denis Zolotarjovs; indictment reveals government connections
Mozilla
Firefox patched 271 bugs discovered by Mythos AI vulnerability discovery tool
Anthropic
Claude AI model vulnerable to DOM injection via Chrome extension; malicious extensions could hijack prompts
Amazon
Data centers targeted by Iran during geopolitical tensions; critical infrastructure vulnerability
FCC
US regulator banning foreign-made routers; initially planned patch ban until 2029, reversed drone patch restrictions
People
Patrick Gray
Primary host discussing security news and trends throughout the episode
Adam Barlow
Co-host providing offensive security perspective on GitHub Actions, DNSSEC, and kernel vulnerabilities
James Wilson
Co-host explaining TanStack compromise mechanics, GitHub Actions risks, and kernel privilege escalation bugs
Bobby Filler
Sponsor interview guest discussing how customers evaluate agentic AI in security products and AI fatigue in market
Niels Provost
Conducted research on using LLMs for vulnerability discovery; featured in Risky Business Features interview
Denis Zolotarjovs
Latvian hacker working for Karakurt ransomware gang; indictment reveals Russian government connections
Joe Cox
Investigated Chinese deepfake software used for scams; documented white-glove service and detection evasion
Zach Whittaker
Reported on Latvian hacker indictment and Russian government links to ransomware operations
John Greig
Covered Muddy Water ransomware obfuscation tactics and CISA CI Fortify critical infrastructure resilience initiative
Lily Hay Newman
Wrote article on disabling Gemini in Chrome; hosts discussed as historical artifact of AI resistance
Ryan Permeh
Early machine learning security pioneer; mentioned as parallel to current AI hype cycle in security
Quotes
"buddy, you're in security now and there's no prize for getting just about everything right"
Patrick GrayEarly in TanStack discussion
"GitHub has given everyone a foot gun and said, don't shoot yourself in the foot with this foot gun"
Patrick GrayGitHub Actions security discussion
"it's 2026. Like, come on. Who did that headline?"
Patrick GrayDiscussing Forbes headline about Mythos finding FreeBSD bug
"infinity minus 271 is still infinity"
Gruck (referenced by James)Discussing Mozilla patching 271 Mythos-discovered bugs
"we're not just vacuuming all this stuff up and then shipping it off to a frontier provider and being like, give me a response back, charge me money and keep the data"
Bobby FillerSponsor interview on data flow concerns
Full Transcript
Hey everyone and welcome to Risky Business. My name's Patrick Gray. Adam Barlow is back on deck and he'll be joining James Wilson and I in just a moment to talk through the week's security news and there's lots of like awful and funny things happening so that's going to be fun. And then after that in this week's sponsor interview we're going to be hearing from Bobby Filler who heads up artificial intelligence over at Sublime Security. Sublime Security makes the most modern sort of contemporary iteration of an email security platform. So if you are, you know, looking to get the best in class email security platform, you want to hit up Sublime. And we're talking to Bobby about, I guess, how customers these days are evaluating AI features in products. It's an interesting conversation. They are very AI heavy, Sublime. And yeah, he's going to walk us through the conversations they're having with customers and the questions they're asking, which seem to be the right questions, if I'm honest. And then we can sort of, we also have a bit of a chat about how, you know, LLMAI selling that compares to selling machine learning based AI, if you want to call it that from, you know, a decade ago. So it's all in all, it's a very interesting conversation and it is coming up after this week's news, which starts now. And look, we've got so much wonderful, delicious chaos to talk about, but we're going to start off by having a chat about this mini Shialud worm. We've seen this worm originally pop up last year sometime. We talked about it at the time. You know, it's a self-propagating NPM worm. In this case, though, the initial access is a really interesting thing, the thing that started this all off, and it wound up infecting TanStack, which is an extremely wide-used thing in the dev ecosystem. I mean, James, you're the engineer among us. We're going to start off with you on this one. What was the interesting vector here? And can you give us a bit of background on TanStack? Yeah, let's start with TanStack because this is complicated machinery in a complicated landscape. So if you're building a React app, you know, React is like one of your two fundamental decisions. React is the framework that you're working with and you're probably writing the code in TypeScript. But that's kind of akin to saying, well, I've bought my block of land and I've got my plans for my house. but there's a heck of a lot of other decisions you need to make about how you're going to assemble that app. And it's things like what's going to handle the API routes, what's going to handle the state management. All these things are an entire separate ecosystem of components that have built up. And TAN stack is a set of those components. They've become wildly popular and they've sort of forged their own paradigm within the React community. So that's what it does. It is a very integral part of building a React app. Now, the thing that is super interesting about the way that this initial attack vector happened here is there was no leaked credential. There was no phished credential. There was none of the traditional sort of ways in that you would expect for an initial compromise. Instead, it relied upon a malicious pull request making its way through a GitHub action. And that GitHub action was to the admission of the TAN stack folks improperly configured. but it's just delicious how they did this. We talked a little bit about this. We ran through the run sheet before we got recording. We do this every week, right? And the misconfiguration here was quite subtle. This was not like Tanstack did something completely suicidal and dumb here. They did slip up a little but why don't you walk us through the mechanics of how this malicious GitHub action would wind up giving these attackers access to Tanstack's repos. Yeah, it is funny, right? Because you pulled me up on the fact that I was approaching this from my software engineering perspective, which is like they tried so hard and they did almost everything right. And you said, buddy, you're in security now and there's no prize for getting just about everything right. And sure enough, there is a particular GitHub action that happens on a pull request trigger, which is one of the most dangerous areas where a GitHub action can operate because it's essentially the moment when someone says, I've got a pull request, a set of changes that I would like you to bring into your repo. and this action fires within the context of the repo that the change might potentially be merged into but can if so configured also pull in those untrusted changes that have been proposed by the external third party in this pull request the advisory from github that's actually been out for a couple of years now to be fair does say this is a very dangerous foot gun if you're using this pull request trigger action. Be very, very careful about bringing in the untrusted part of the code repo because this is what can happen. And they did not follow that advice. And that's exactly what this GitHub action did. And that's how the attacker got their foothold. They got code run. They poisoned a cache that was then used during the deployment. And it was actually the legitimate deployment step that then pulled that cache out and resulted in the bad binaries being uploaded to NPM. I mean, it does feel a little bit, though, that GitHub has given everyone a foot gun and said, don't shoot yourself in the foot with this foot gun. You know, Adam, let's bring you in on this one. I mean, this GitHub action stuff, it just seems like perhaps GitHub could be doing more, but I don't know because I'm not an expert in this field. Yeah, they absolutely could have avoided shipping people a foot gun. I think kind of here what we are seeing is that I don't know that anyone really expected the whole industry to coalesce around GitHub as the way of building everything. And, you know, one of the things that I thought was interesting about the story is that. Well, remember, sorry to cut you off there and ruin your flow, but you remember when Microsoft bought GitHub and everyone's like, it's over for GitHub. That's it. No one's going to use GitHub anymore. GitHub's dead. right yeah no it's it's weird how much github has become you know a critical part of everyone's of infrastructure and flow um but yeah so in this particular case i feel like like they did so many things right as as james was saying um but the github action kind of set up like this i don't know that anyone really expected this to become so important right and this particular foot gun to be such an important thing i mean the idea of running a little bit of automation when you push stuff in and out of a git repo or make pull requests or whatever else seems like a good idea and all these little things they all seem like good ideas until you start really crossing important trust boundaries and then building infrastructure that in very kind of nearish real time pulls in dependencies from other people and so on and so forth and we end up with the impact of these you know kind of little choices turning into, you know, quite widespread compromise of code. I mean, I thought the trick of using the kind of shared cache between, in this way, the GitHub Action can cache artifacts that they've built or whatever else they want to speed up later steps in the process in which when you are re-downloading, you know, heaps of infrastructure or heaps of dependencies and stuff like that, that can make a pretty material difference to how responsive your build pipeline feels. And developers love having their build process be really snappy because then they can turn things around really fast and it feels like you're doing stuff and you know that uh you know kind of desire for everything to be snappy and responsive was the second kind of part of this right i mean the first thing is github actions you know executing kind of macros i guess or triggers automation events that potentially have untrusted inputs one problem you know the side effects of those untrusted inputs being processed in this case poisoning a cache that's then used later on is the second part of the puzzle and then the third part of course is once it was executing in the release context then it drops this worm that will propagate into other people's repos steal creds well let's let's you know let's go there because we've covered the tan stack part of this but then from there as you say it dropped a worm which went off then and self-propagated and it's like it you know you just get on social media and it looks like people are having a bit of a hard time containing this thing like it is running real quick you know and we went through this last year and everyone's like oh we've all taken steps to slow it down and GitHub's like, oh, you know, we're going to, this ain't going to be a problem in the future. And like, here we all are. And this time, the people behind this worm have added some real nasty features. And one of them is if you try to rotate cred, so it will, you know, hand off all of its API tokens and stuff on your machine to the attacker. If you try to rotate them, it detects that they've been invalidated and RMRFs your whole drive, which is just like the nastiest thing to do. I don't even know why you would do that unless you're just like, I don't know. It's like a real anti-social personality disorder kind of thing to do with your NPM worm. But like this thing's out there now and causing all sorts of drama. But to what end is this, you know, sitting above this whole thing? Is this going to be something really dumb like someone trying to steal cryptocurrency or something? Because that's the vibe I get here. It's very hard to tell because, again, this all amounts to stockpiling creds that we then go like for what and for where. And so similar to PCP, it might be a little while until we see either the follow-up actions that they decide to do or they just kind of farm the creds out and let other actors have a go with them. But it is odd that we don't know where this is going, but it'll go somewhere. The other thing I wanted to add in here is I had a real cold sweat moment of panic this morning when I read this because I thought to myself, it's okay. I haven't done like an NPM install or a bun install for a while now. So I know I didn't pull in packages during this, you know, six minutes, six whole minutes where this was a blaze. But then it dawned on me that the way that coding agents work now, both Codex and also Claude, they work in Git work trees. And so when you farm your agent off to go and do a task, it's working in a new directory that doesn't have all these modules installed. And so time after time after time throughout my day as I'm interacting with agents, they are pulling these packages over and over again. So, yeah, folks need to think, you might not have done an NPM install during this window, but which one of your agents was working on something that went and did it? Yeah, you might not know that you did that. You might not know. That scared the hell out of me this morning. Yeah. All right. We'll talk about our incident response later. But, yeah, probably time to move on to the next story now. And funnily enough, the next story we're going to talk about is this breach of Instructure and the learning platform Canvas. which is used by K-12 schools and colleges worldwide. I mean, it's reported as being an America thing, but I can tell you that, you know, universities here in Australia and across in your country, New Zealand, Adam, have been like delaying exams and things and dealing with this. Funnily enough, though, it being the big story, I don't know that we've got much to add here. I mean, it was a small breach that initially instructor was like, oh, yes, you know, attackers tried to compromise us, but we have contained it. And yeah, it turned out not so much. A lot of their data got walked, including, according to the attackers, billions of messages between students and their educators. And then, of course, there was a ransom note kind of dropped on a shiny hunters ransom note dropped on the login page for this system. Reporting suggests that they have actually paid now to get the data deleted. And, you know, I mean, that's pretty much the end of it. It looks like shiny hunters managed to rack up a win here. Adam, any thoughts? Yeah, I mean, it's just, you know, paying them out feels bad. But on the other hand, you see the amount of pain it was causing. You can understand why that was a decision that they were going to consider. And especially in cases where you've got a supplier and their customers, and the customers are all applying pressure to the supplier to do something, anything, and the only fast option that a supplier has is to go pay. And that feels bad, but you understand why. And, you know, I don't know. Hopefully very little of it actually ends up in the pockets of a shiny hunter. But, you know. Well, you hope they slip up doing their money laundering or something, right? Like that's the vibe I get here is that that could well be how this ends. Because Shiny Hunters definitely has like UK teens vibes, right? Yeah, and I can't imagine their money laundering slash, you know, kind of money handling slash not going out and immediately spending it on trash that's immediately kind of draws attention to them. doesn't feel super likely. Like they may be good at the tech stuff, but that doesn't necessarily translate to good at, you know, long-term crime. I mean, everyone always criticizes people who pay, but I've said it on the show a million times, right? Like sometimes it's existential and they've got to pay. Like, and it's really hard to say that it's uniformly wrong in every circumstance, which is why I was against legislative proposals that would have outlawed paying ransoms. Yeah, I mean, I guess that's why I cast it as feels bad as opposed to is the wrong thing to do, you know? Yeah, exactly. It just feels gross, but what are you going to do? Well, we're going to stay with you on this next story, Adam, because this one has you written all over it. I thought of you as soon as I saw this. Where to begin? So someone screwed up in Germany, rotating a key signing key for DNSSEC, which meant that the entire .de TLD was returning servfail on DNS queries where they were signed zone files, right? So they broke the chain of trust for all of .de. And Cloudflare, basically its decision, and I agree with this decision. I think it was the correct decision. And apparently there's an RFC backing this decision as well, but they just let the whole thing file open, right? So basically they just switched off for the 1.1.1 Cloudflare resolver. They just switched off DNSSEC validation for all of Germany because of this screw up. And I think, look, absolutely the right decision, but it kind of goes to show you that like, this isn't really news. You know what I mean? Like I'm reading this straight off the Cloudflare blog. The fact that an entire TLD and a big one at that just turned off DNSSEC validation and everything was like, no one noticed. Kind of tells you what you need to know about whether or not the DNSSEC juice is worth the squeeze in my view. But I really came for your opinion on this one. Yeah, it's a pretty interesting tale, this one. And I think you kind of summarized the guts of the technical aspects. The interesting bit with Cloudflare is their 1.1.1 resolver does enforce DNSSEC and validates the domains that it's answering questions about. And the correct behavior in the situation was to return an error, right? Return serve fail and not answer the query. and Cloudflare very rapidly realized that, you know, that was worse than just answering the queries and marking them as not secure, you know, saying like, here's the answer, but we're not, you know, it's not DNSSEC validated, which is what they ended up deciding to do. Well, and thankfully, Adam, thankfully, all of the software that we use and rely upon out there is set up to really take note of that note in the return zone file that says this isn't secure, you know? And, you know, this really changes a lot of things. not we but i know we played with a bunch of dns stuff over the years and um you know making the like it's one thing to sign your zone files and publish it for other people to validate it's a whole other thing to say we are also going to make all of our queries fail if dnssec isn't available or you know should be available and isn't because it's just gonna break stuff and the amount of breakage that dnssec causes versus the amount of you know kind of impersonation or you know cache poisoning or whatever other things it's trying to prevent really the you know the impact of dnssec is mostly about bad availability and not about integrity and yeah does the juicy worth the squeeze for dnssec really not and especially now that we have so much other crypto layered over the top with tls let encrypt one let encrypt one and the thing is like i i guess one of the reasons that i like to beat up on dnssec is the proponents of dnssec like the real rabid ones are among the most annoying people you'll ever meet in your life right i mean dnssec clearly grew out of the kind of cypherpunk way of thinking right where we should make it perfect uh without really accepting the reality of the world that we have to live in uh and yeah it's I mean, DNS itself is just old tech, and then bolting crypto into old tech, you know, brittle is the end result, right? We have Let's Encrypt. We have modern browsers. Let's move on. Yeah, and it's not perfect, right? I mean, the Let's Encrypt world and the browser, like, delegating this all out to TLS, you know, isn't the best solution, but it's the solution we've got, and it's the only one that's, you know, kind of viable in the real world. I'd argue, Adam, that actually, in that everybody uses it, it is actually the best solution, unlike DNSSEC, which people don't actually use. But anyway, we can argue this one more at great length over a beer one day. Moving on, and Google's threat intelligence group has released a report all about what's happening out there with adversaries and whatever. And not surprisingly, AI features very heavily. I guess the item that they spoke about here that's been talked about most is they discovered some threat actors had used AI to uncover an ODA in some sort of web IT administration tool. I don't know. That sounds like cPanel to me. I don't know. But it's like an MFA, something like that, I guess. But it's an MFA bypass bug that they found. And they were able to, I guess, disrupt the actor from being able to do widespread exploitation against that. So that's great, a wonderful win for Google. But the thing that's remarkable to me is looking through the executive summary of this report, and it's all stuff we've been talking about, like at length, on the show for months and months and months. James, you've been through this one. That was your take as well. Yeah, exactly. You know, they open with that exact statement that this is really just a trajectory from nascent AI usage by attackers, which is where we were from the last report, to now this is, you know, to use their term, industrial scale application of generative models within adversarial frameworks. But the nice thing about it is they sort of break it down into six headings, and each of them is a very sort of targeted look at where AI is being used for vulnerability discovery, AI augmented development of defense evasion, autonomous malware operating end-to-end with AI. There's a good section in here as well about the obfuscated LLM access, and I think that's something that needs to get a whole lot more attention throughout industry is just like how do we prevent the large scale use of LLMs in an unauthorized sense from these bad actors through things like, you know, chat interfaces or other half-baked LLMs being shoehorned into this product, then it's accidentally a really nice distillation vector for an attacker. But overall, it's not a super thrilling report, but it's just really great to see this all condensed down in one place that says, yes, this is happening and the trajectory is we're now at industrial scale and let's see where this goes from here. Yeah, I mean, Adam, you would, as someone who has spent your entire career basically working in OffSec, I'm guessing you would have found this one pretty interesting. Yeah, yeah. It's a good summary of kind of where things are at and all the various places you can use the tooling. And, you know, I think much like James, the idea that we can control access to models as a viable kind of strategy for mitigating the various ways that it's being used doesn't seem like a really robust path forward. but I thought it was just you know this is a great roundup of the state of the world and obviously they have insight you know by virtue of being both incident response but also operating one of the big models and they can look at the arts being used and so on so yeah it's always interesting reading their work because they have that kind of both ends perspective on it. Yeah and if you want to know about the state of the art in terms of using LLMs to do vulnerability discovery last week we spoke about some work from Niels Provost who's an old school you know security head who did some work in instrumenting and orchestrating LLMs to do Volndev in a way that was like as effective as Mythos, even using local models and older models. That was some very interesting work. We did talk about it last week, but since then, James did a 90-minute interview and discussion with Niels all about that work, which is available in the Risky Business Features feed. So again, I know I've been banging on about it every week, but if you are not subscribed to that feed, you are missing some really good stuff. So head over to either risky.biz to get the links, the subscribe links, or you can just fire up your podcatcher and search for risky business features. But that is a fascinating discussion. And I've also linked through to it in this week's show notes. Now, a big thing that happened, Adam, while you were away for a few weeks is every week, it was like the agenda seemed tailor-made to the special guests that we had that week, right? just incredibly well tailored and then you come back and it's the same right because we got a whole bunch of really interesting bugs to talk about in stuff that you know very well so first of all there's the dirty frag bug we spoke about copy file last week and James I believe you're going to correct something that you said last week that was incorrect about that but there's this new one called dirty frag we've also got bugs popping up in like free BSD and whatever a lot of This feels very AI driven. And I just wanted to get your thoughts on these bugs, Adam. I mean, I really enjoyed Copyfail. It's a beautiful bug. And I went through and read some of the coverage of that when I got back from my holiday and wanted to refresh my memory about what the cybers was all about. And it just, it felt so familiar in so many great ways. Dirty Frag is essentially just another variant of the same bug. The guts of Copyfail were that you could write, corrupt the disk cache in the kernel. You can overwrite data stored in the kernel's idea of cased files off disk. And that was done through, in that particular case, something in the encryption plumbing somewhere. This is another couple of vectors in the kernel that you can use to write to the page cache in ways that are surprising and use that for local priv-esque. And they're beautiful local priv-esque bugs, right? You don't have race conditions, you don't have memory corruption, like it's targeted, repeatable, it's exactly what you want in a kernel local Prevesque because the last thing you want is bugs that are going to cause instability. You want things that are super reliable. And so we love a Linux Prevesque. We've had to think about Linux being essentially single user from a security point of view probably for the last 20 years. It's never really been safe to have multi-user Linux boxes and it's nice to see that sort of reinforced for everybody. But yeah, I enjoyed both of these bugs. Actually, Dirty Frank technically is kind of two different instances of a repeat of the bug. That one that works well on Ubuntu, one that works well on everything else. But yeah, they're well worth reading and understanding and I felt, yeah, it felt nice seeing something so near and dear to my heart on the run sheet this week. Well, and then there's the free BSD one too, which is also pretty hilarious. And I mean, these are all AI discovered bugs. As far as I know, I think this FreeBSD one was a Mythos discovery. And I think one thing when I was doing a little bit of research on it, I plugged it into Google or whatever. And I think Forbes was running a story saying Mythos has found a bug in one of the world's most secure operating systems. And I'm like, man, it's 2026. Like, come on. Who did that headline? But yeah, walk us through this one as well. Yeah. So the FreeBSD bug absolutely feels AI discovered. And it feels a little unsporting, honestly, letting an AI look at 26-year-old or whoever old this, you know. But it's the most secure operating system in the world, according to Forbes. They may have confused their BSD variants there, perhaps. Anyway, the particular bargain question is that a malicious DHCP server can set a value that gets written, the set of value that's given to the DHCP client, that when the free BSD DHCP client writes it into a cache file on disk for later use. You can kind of, you know, there's incorrectly escaped meta characters. You can inject, you know, more directives to the HTTP client. And then next time it runs and reparses that file, it interprets those directives and you get code exec. And the, like, that feels like an AI discovered bug because, you know, you know, sort of chaining that logic together of how you would use it and what it's good for makes a lot of sense. The thing that I really liked though is this bug, so writing into the lease cache file on disk, Dirty Frag writing to the cache files, writing to the cache in memory, and that bug way up front with Shai Hulad, all three of those are cache poisoning. And I'm reminded of that like classic amorphism about there being two hard problems in computer science. One is naming things, the other is cache invalidation. And that definitely felt like, yeah, that is ringing true this week for sure. Yeah, yeah. Yeah. Meanwhile, James, you wanted to correct something. When you were talking about copy fail last week, you said you got one of the technical details wrong. Yeah. So when I read the publication from the Theore folks, there was mention of using the, what was it, the IPsec encrypted sequence numbers. There was a bug in there that basically resulted in a predictable four byte right outside of boundaries. And I assumed that that meant that that was just a really simple like buffer overflow in that. And on one hand, you think, okay, neat that they found that but you also think there's so much tooling and and stuff that should have caught things like that um and then so we'll i think we'll link to it in the show notes but there's a great write-up from retro.zip retro with a zero where they actually sort of did a bit of a record scratch of like it's not your average four byte right out of bounds and they go deep into this and it is crazy the the level of sort of hoops that were jumped through to just get this four byte right into the page cache. So superb work and glad they took the time to really explain it. Yeah, that write up is absolutely worth a read if you want to understand the specific details, because like it explains it so well. I definitely recommend that one. Yeah. And we've linked through to that write up in this week's show notes. Now, look, we're talking about AI discovering flaws in other things. There's a fix just gone out for the Claude Chrome extension, which would have enabled other plugins, I presume they mean extensions by that to hijack the Claude extension there. Is that about right, James? Yeah. You look at this one and you just go, Claude does not belong in a Chrome extension right now. Because it's just so simple how this was done, right? So if you've got the Claude extension running, someone else, or sorry, someone could load another extension into the browser that had no permissions, no elevated permissions whatsoever but just had some nasty code in there that was interacting with the page. And then if the browser goes to claw.ai, for example, the malicious extension, all it has to do is just inject something into the DOM, which is the bread and butter thing that all of these extensions do. And then clawed sees that thing that's been injected into the DOM and reads it as just a prompt. It's like if it's this easy to trick clawed into reading a prompt out of the DOM in a browser, get that thing the hell out of an extension like the only safe way to really use a model these days is for you to be the sole source of input into the initial prompt right that that is like the the cleanest guardrail and surface we have at the moment is because that's when you as the human express your intent your instructions yes we pull in a bunch of skills and other things along the way but you've bootstrapped that you've set the task and that's generally what the model will follow but when you take Claude and put it in an extension that is reading the DOM and that can become the prompt nothing good is going to come after this my friend it's gonna be that's gonna be a bad time and meanwhile Google copping a bit of flack this week for shipping everybody a four gig like local version of Gemini with Chrome where really people just woke up and their computer had this seriously just downloaded this four gig update funnily enough I mean But look, this is to be expected, right? This is the way the world is going. And I've included a story in this week's show notes from Lily Hay Newman over at Wired. The headline is, you can disable Gemini and Chrome if it's freaking you out. And the reason I really wanted to include this is this is going to be a future historical artifact, this article, where people will be like, wow, people thought they could avoid AI. It's like saying, you know, here's how you can have a clean install of Windows and disable all web browsers, right? Like you don't want to use those web browsers. In a browser, you used to be able to turn off JavaScript and that was a legitimate decision that people would make. It's the same sort of lineage. Yeah, exactly. But I mean, what do we think about Chrome doing this? Adam, I'd love your thoughts here. I mean, I think the argument for having that tiny model in there was actually pretty reasonable. Like being able to do certain things that you don't necessarily want to shove off to the cloud. Obviously Google, by making you run the model, even with the really little ones, saves them quite a lot of compute, I would imagine, rather than having to have safe browsing, make a call off to a Google AI service every time someone visits a page. No, no, no. So your argument is reasonable, but the counterpoint is, witch! It's a witch! Yeah, I mean, the amount of things that are going to have like AI models stuck in them, like it's, as you say, it's just like, you know, we may as well say, let's not put DLL files in things anymore. You know, that's the kind of the level that we're at these days. So like, I understand that some people have, you know, concerns about AI just generally as a concept, like that the training of those models in the first place was unethical or whatever and that's kind of like i can respect that kind of point of view of it but if you're a technical user and you want to turn off this piece of functionality in your browser you're kind of on a losing wicket i think yeah i mean i think there were more solid there were more solid arguments you remember when sony started putting like basically malware on its music cds back in the day where if you played one of their cds it would like trojan your box to put all of this crazy like you know kernel level like anti-copy stuff on your computer without asking you. Like, you know, that one, okay, I think that we could say that one's over the line, but like a browser shipping a model, get used to it. Yeah, pretty much. Yeah. Now look, we've just got to like, I got to rub our temples. Now we've talked about like AI bug discovery. That's a very big deal. And we've also talked earlier in the year about like how AI is being used to orchestrate attacks and scale them up and whatever. And, you know, while we're on that topic, bugs in Palo Alto and Avanti still I mean this is the sort of stuff that's gonna get absolutely auto owned by orchestrated AI agents which I am now referring to as infinity e-script kiddies because that's basically what these agents are but my god man like at this point you know running stuff like Avanti running stuff like a lot of the Palo Alto gear it's just you just it was risky enough before and now it's just suicidal yeah i mean the you know they're being yet more bugs in a vanti endpoint manager like how is that even how is there any code left that is like surely they must have got rid of all of it by now there can't be anything left um but i don't know what this well as as gruck said and you you weren't here and i doubt you listened to it because you were on holiday but as gruck said here a few weeks ago when talking about mozilla patching 271 bugs that were found with uh with mythos in firefox he said infinity minus 271 is still infinity and i think that just applies here i think i think so yeah crack crack with the wisdom as usual um yeah i mean ultimately the thing that stood out to me about the avanti story was the seesaw of avanti came out and said look we just want people to understand that we are trying to do the right thing it's like nobody the time to do the right thing was 30 years when you stopped investing in the security of this product, all the people who bought it, who bought it, who bought it, you know like many corporate acquisitions ago Like that the problem is that we running 30 year old trash code and expecting it to be robust against modern internet modern AI like much like that 3BSD DHCP client being unsporting as a target. Like, at this point, Avanti, like, it's just, like, it feels like you're kicking a puppy at this point. Yeah, well, and this Palo bug too, Adam, is quite awful, right? So I haven't seen, I was looking around for a POC. Apparently there is a POC. So this was a Palo Alto remote code exec. it appears to be memory corruption in a content length header in the year 2026 in a thing that parses web content for a living i mean that's babby's that's babby's first exploit really it really is um so there's a slight kind of mitigating factor in that this is in their like captive portal bit which there's probably no reason to have internet facing but of course people will because you know why wouldn't you uh but yeah come on like memory corruption in a security clients in the year 2026. And like, there should be defense in depth and exploit mitigation and all these things, but security appliance vendors haven't had to update their products for 20 years. And so, you know, why Railroad and Pi and do all of the other exploit mitigation stuff in here that would let you get away with having mem corruption in your content network? But no, that's just Palo Alto life. So everyone's at least used to patching this stuff. So that's good, right? Yeah, James, you look like you had some feelings there. I just, to that point, everyone's used to patching it. But the thing that I got a good giggle out of was when I read the Avanti advice, it sort of is, they bifurcate what they tell you to do based on how well you responded to the last time you got owned on this box. So it's like, you know, if you responded correctly to our advisory in January and rotated your credentials, you need to do these steps. If you didn't, you are in this bad state. And if your box wasn't compromised in January, you need to actually do this instead. So I just love that it's now like you've got to look back over your history of, you know, not just what do you do with this box now, but. So they released a choose your own adventure to accompany their advisory, basically. Decision tree of fail. It's great. Moving on. And Russia is launching kind of its own version of Starlink. It looks like there's a great write up of this in Wired and we've linked through to it in this week's show notes. But it looks like it's going to take them a while to get this thing up to being quite reliable. But even an intermittent satellite connectivity over a battlefield is actually going to be quite useful. But what's amazing is like how quickly this sort of capability has been understood to be very important by major companies. I mean, we've got the Europeans essentially launching their own thing and doubling down on their own thing because they're a little bit worried about continued access to Starlink because of Elon Musk and the United States. its attitudes towards various things. And no doubt the Chinese will be working on their own version of this, but it just seems like, you know, this is absolutely a very important capability and they're going to do it. I mean, there's some delicious details though, in this write-up about how the hardware for this Russian version of Starlink is like multiple times bigger than anything that comes out of SpaceX, right? Which shouldn't be all that surprising, but you know, they're giving it a crack. James, I know you've been through this one. And I guess one of the things that you and I both zeroed in on is the orbit path for these satellites is quite different to those from the other companies, which I don't know, there's some, that could be interesting in terms of, well, could you shoot these things down and have it not actually damage your own satellites, for example? Exactly. Yeah. There's two very interesting differences. One is just the count of the satellites, you know, whereas Starlink has thousands and thousands of these things up there to get full global coverage especially around densely populated areas the plan here is to only launch well only 300 satellites by 2030 that alone will require them to churn out a satellite or two a week which seems a little bit ambitious but so it's a much smaller number of satellites but the the orbit path because they want to essentially cover you know russia and and its various territories around there they can get away with an almost polar orbit on a certain incline that means that they're going to be out of the way of the other satellites and they're also operating at a much higher altitude about 800 kilometers versus 500 kilometers so it sort of settles an argument you and I were having because I said you know there's no way that that you know an adversary would shoot one of these down because you take out all the satellites and then you know we'll never we'll never go through space travel again but you know if it's only 300 that doesn't sound like you actually need to disable too many of them And even if you had to do something kinetic, it's probably in a flight path where that debris won't be too dangerous. So, yeah, gosh, when everyone's got these, it's going to be interesting to see what happens if there's a kinetic conflict around them. Well, the next world war is going to happen in space, I guess, as well as everywhere else, I guess, is where we landed on this. But I think also, you know, there's probably ways to disable these types of satellites without creating a debris field, whether that's just putting a hole in them with a laser or burning out some of their sensors or antennas. You know, there's probably ways to do it. And no doubt the very smart people over at US Space Force are working on exactly this problem right now. But yeah, just an interesting sign of the times. Moving on, we've got a report here from TechCrunch from Zach Whittaker, which is looking at a Latvian hacker named Denis Zolotarjovs. This guy was doing ransomware stuff, working for a Russian ransomware gang called Karakurt. the interesting thing is here though that the indictment sort of spells out links between this group and the Russian government basically so it's really good to have some of those links spelled out in a more explicit way than we've had previously and that's just why I flagged it and put it in this week's show notes we've also got a report from John Grieg over at the record talking about how the muddy water crew which is an Iranian APT crew essentially they've been dropping chaos ransomware to kind of cover their tracks, which I don't know, that's not so surprising. I mean, it's right there in the name, you know, muddy water, trying to muddy the waters on attribution. Although, James, you pointed out to me that this does not look like a particularly, you know, effective way of obfuscating attribution, given that they're signing utilities with certificates that people know they use, for example. Yeah, it did seem like well-intended, but perhaps poorly executed. But I think, you know, you made a good point to me, which is that even if you get to the point of realising, huh, this ransomware was actually signed by, you know, Muddy Water, you've already had to deal with that ransomware and all the impact. And that's what they're aiming for, is just to slow you down, to distract you, so they can get on with doing what they want to do otherwise without being noticed. so yeah um yeah i feel i feel like this will actually work as an obfuscation against people who don't call mandiant which is most people but if you do call mandiant they're going to pick it apart and figure it out i mean is that your vibe too adam yeah yeah pretty much but i think like the the specific value of this you know in in you know in terms of it being believable or whatever it's kind of less important than you know the goal is to get people onto the ransomware off the playbook right they might have a playbook for how to respond to ransomware if you can shunt them over onto that you know what they're going to be doing for the next three days and it buys you time and like anything where you can like manipulate how your adversary well how your victim i guess i'm still thinking my off sick life um anything where you can manipulate how they respond gives you predictability and that's just really important it doesn't have to hold for long it might just be long enough for you to you know action on objectives or whatever else so yeah i think it's you know it's still a worthwhile trick yeah and uh speaking of ransomware fox Foxconn has confirmed there's been some sort of attack against its factories in North America. Looks like a limited impact, though. I mean, from what we can tell so far, it looks like they had a bit of drama with, you know, the Wi-Fi didn't work and they couldn't use their computers and whatever. And I don't know if that's ransomware or if it's people like, you know, if they're pulling stuff down and responding or whatever. But it looks like they're all back up now. Many such cases these days, right, where you actually see, you know, there's an initial foothold. There's a bit of drama with a ransomware attack and then things are quickly brought back to normal. So it looks like perhaps that's what's happened here. But one story I wanted to talk about in a bit more depth. John Greig has this write-up in the record. There is a CISR initiative where it's called CI Fortify. And the idea here is that they're going to help critical infrastructure operators to figure out how to operate offline. Now, this could be because there are DDoS attacks happening targeting critical infrastructure. It could be because a campaign like Vault Typhoon is starting to do unthinkable things to US critical infrastructure. Could be for any reason like that. But I think this is a really good idea. I think in terms of like resilience, you know, going to a utility, going to a bit of critical infrastructure and saying, look, how dependent are you on US East 1, right? And that link being active in order for you to actually be able to operate your service. And like, can we build some contingencies here? funnily enough james at one point in your career you kind of went through this uh with a critical infrastructure provider and where it's landed is okay so for the for the really insecure utilities and whatever going through this process is is going to be a really good thing to do but you made an interesting point to me which is for the people who really know what they're doing for the highly sort of secure environments having to go through an exercise like this actually winds up introducing a fairly large amount of new attack surface because there's all of a sudden a lot more new equipment and a lot of redundancy. So yeah, there is a downside, I guess, for some operators was the point you were trying to make to me. Yeah, I was looking at it from the perspective of if an organization has gone through their cloud transformation and moved a lot of on-prem workloads up into the cloud and they're heavily dependent on telco infrastructure that is managed for them, That's the kind of stuff that is straightaway is going to be in the crosshairs if they're challenged to say, well, how do you operate if all of that stuff is gone away? Inevitably, and this is the situation I went through, is we had to start bringing some versions of software, really key critical things back on-prem in a hybrid cloud and on-prem model. Now, that introduces complexity, right? not just it's one thing to go from cloud to on-prem but if you're having to manage both active standby or active active on-prem and cloud at the same time you've got complexity you've got additional equipment you've got additional configuration and all of those things when you're not in the offline mode become really excellent places for an attacker to get access to and to dwell my point is i think this is still i mean you have to go through this right is would be my point which is we live in a world where these links these data centers i mean we saw in iran right iran actually attacking amazon data centers for example right when there's going to be a conflict these this infrastructure is going to be targeted and we need to make sure that our critical infrastructure is resilient yes it does cut both ways but i don't really see that we have another another option but to do this adam what's your take on this i think this kind of exercise is really useful and important because you know we build this stuff so quickly all this infrastructure where everything is all high tech very quickly and we haven't really thought about the failure modes now what you do with your if you go ahead and do this kind of exercise and you understand now what the potential failure modes look like what you do with that is another thing right building redundant infrastructure or offline bringing stuff you know back out of the cloud like those are things paths that you can go down but at least understanding what the potential impact could be so that you're not exploring that the first time when it actually happens and i think in New Zealand for example we have one essentially one or two kind of big fibre links in and out of the country if those go away because someone does a little submarine snippy snippy which you know great powers have been known to do in times of conflict little snippy snippy and like we don't understand and like sure there's a bunch of things we could do it's probably prohibitively expensive we're just going to accept the risk but at least understanding that you know all our national payment systems are going to stop working without that piece of fibre like that's good knowledge and it's worth doing these kinds of exercises to know, are we going to have power? Are we going to have water? You know, because you don't want to be doing this for the first time for real. And of course, this will happen because the first objective of a superpower during a great power conflict will be to take New Zealand out of the war. Strategic dagger pointed at the heart of Antarctica. Yes. Now we're just going to stay here with our sheep. And because we're not exporting all of our food anymore, we're going to have plenty. It'll be fine. I hope. Just no diesel. Yeah. What is it? The New Zealand defence strategy of, well, they have to go through Australia first. Thanks, buddy. Your taxpayer dollars buying F-18s, thank you very much. F-35s, thank you very much. We get the good stuff. I wanted to link through to a report, which unfortunately, unless you're a 404 Media subscriber, you won't be able to read it on web. If you are an email recipient, you would have received this one, so you can dig it out of your inbox. It did go out for free as an email. But Joe Cox managed to get his hands on the software, a bit of Chinese software that's used by people doing scams to do deepfakes, right? So you can grab pictures of people, send them to these guys for 500 bucks. They will create like a model of this person you're trying to impersonate. And it will do it in real time on Zoom, on WhatsApp, on whatever you want. a really fun write-up of like Joe's adventure in like how going and procuring the software and they even like remotely set it up for him they set up a partition on his on his computer and in they came remote support very slick operation James I mean you know I know you enjoyed this one as well I did a couple of things really jumped out at me first is yes just it was such a white glove high touch service so it's beautiful to see they've considered customer service but didn't take a lot in terms of hardware specs. They demanded a i7 processor 16 gig of RAM and an NVIDIA 4080. That's not the kind of spec that I thought would be needed to pull this off. So that was quite surprising. But the thing that is like just this moment when you go, oh my goodness, we're in trouble, is when you read the part that says, you know, exception spelled with an X, which is a deepfake detection model. It struggled, they say in inverted commas, but struggled was actually almost 100% of the samples from this software they acquired was mistakenly labelled as authentic, despite this research being the state of the art of deep fake detection. And the videos look good. There's still a little bit of uncanny valley, but the fact that it works when there's something in front of the face, it tracks lighting differences, gosh, it's getting good. Yeah, I wonder. I wonder. I mean, we've had Persona on a couple of times, right? And they talk about how they do real-time video and stuff. And that's a big part of how they actually do like KYC style, you know, verification of identities and it's being used in the enterprise and whatever. And you worry how enduring that sort of approach is going to be. Now, I'm sure they've got labs and they're cooking up all sorts of ideas and detections and whatever. But this, you know, remotely verifying someone's identity is correct is a wicked problem. It has always been a wicked problem and it's going to remain a wicked problem. And I feel like we've had an easy run of it with video recently and now that's kind of gone. And our final piece this week, which is, you know, I guess we'd call it our skateboarding dog. We spoke about how the FCC in the US is going to ban foreign made routers and it looks like they're pushing through with this. But it looked like they were also going to ban foreign routers from patching and issuing patches from like March next year. They've now realized this is not a great idea. And they have pushed that patch ban out to 2029. Still a bad idea, but it's further away. They also reversed the ban on patches for drones So well done FCC So many bad ideas but yeah bad idea two years down the track we got a chance with it It the bad ideas this week that these days we are reduced to struggling with. So I guess good news and we'll check in on a couple of years and see what they do then. Yeah. Right, guys. Well, that is it for this week's news. Great to have you back, Adam. James, great to chat to you as well. And yeah, we'll do it all again next week. Yeah, we certainly will, Pat. I'll see you then. Yeah, thanks, Pat. See you in a week. That was Adam Boileau and James Wilson with a check of the week's security news. Big thanks to them for that. It is time for this week's sponsor interview now with Bobby Filler, who heads up AI over at Sublime Security. If you are not familiar with Sublime Security, it is the modern whiz-bang AI-enabled secure email platform or email security platform. So, you know, if you need to filter out BEC, if you need to filter malware, phishing links, things like that, It is that sort of platform. It is the most modern iteration of one of them. It's also highly inspectable. You can write custom rules for it, or you can just get their AI agents to do that for you. That stuff actually works really well, crazily well. In fact, but Bobby joined me to have a bit of a broader discussion about AI in the cybersecurity marketplace. When people are evaluating cybersecurity solutions that use agentic AI, what sort of questions are they asking? What are the things they want to know? And let's just start it there. But we go on to talk about a few other things, like how this is a bit similar to the machine learning craze of like 10 years ago. It's a fun interview. So here's Bobby Filler talking about how customers go about evaluating agentic cybersecurity platforms. Enjoy. Yeah, I think they go about it a few different ways. And honestly, the easiest one is just asking questions, right? How has this agent been trained? what is its background of knowledge? Do you use evaluations, offline, online evaluations to monitor performance? When we're talking about things like agentic use and autonomy, has it been red teamed, right? Like that's a real, real situation that folks need to consider at this point is these agents can do a variety of things. They have different skills, different tools they can reach into. And if that hasn't been thoroughly tested internally and externally, customers are understandably wary of that. And then as you move down the line, I think it turns into, well, what's your methodology? What is the reason for building this agent in the first place? Like, what problem did you identify where you felt like I needed this? And those types of questions, I think, really suss out whether or not a vendor is bolting on something that is just kind of an afterthought, checking the box, or whether or not they're building it with good intentions. This idea of up-leveling the customer, giving them an opportunity to grow with the product until the point where they feel comfortable releasing some of their day-to-day responsibilities to it. Now, do you feel like there is, and I definitely feel like this is something that's happening out there, do you feel like there's some AI fatigue among buyers at the moment? because I feel like everybody has like bolted some sort of AI function onto their thing. They're like, we're an AI platform. Cause like, that's what you have to do at the moment, right? It's just what you have to do. So I feel like if I'm a buyer at this point, I'm like, oh, you want to pitch me your AI solution, do you? Oh, great. Like I haven't had like 10 of them this week. Yeah. It's, it's interesting. So I grew, I kind of got my background in the, in the heyday of early machine learning being introduced into security products. I'm from the end game. A lot of my security AI friends were in like CrowdStrike and Silence and things like that. It was always really funny because you would go to RSA, you'd go to Black Hat and be like, I'm the person who, you know, uses math to catch malware. And they're like, no, just like, and you probably remember snake oil booths and things like that popping up. And it was kind of a joke for a while. And it was a it was a tough one i mean it always it always worked though and i think i think it's interesting that you mentioned this right because like ryan perme i know still i'm in touch with ryan who was a co-founder of silence and i remember running into him at black hat once like just before they launched that product and i'm like hey man you know how you beat is like i've been working on something it's really cool uh and silence you know look in the end they had an exit to you know wherever they wound up and it wasn't really that spectacular but the product was interesting i think where they messed up is they missed the edr train right but as an anti-malware engine doing machine learning classification man it worked well like it worked really well the problem with all of that machine learning stuff was always going to be the edge cases it was like how do you handle like anti-cheat on that ships with a game or how do you handle some enterprise products that look like trojans but but that was the thing is fundamentally this technology was incredible but then all of a sudden everybody's like it's got machine learning right it sort of does feel like a repeat of that whole it does i feel like it kind of like ebbs and flows and it's fascinating now when i'm when i'm in a lot of these these customer meetings and and talking to folks i don't get i don't feel the same pushback that i used to in like 2016 what i get instead is there's usually some pressure from a higher up being like look if if you get funding for this project we need we need the latest and greatest latest and greatest is ai and it's like okay so there's some self-fulfilling prophecy there that kind of takes place so people are slapping ai on stuff so that like people can get authority to buy it because they've been there's a mandate from heaven that says that they need to find efficiencies yeah yeah and then i think on the flip side like in 2016 people weren't using even accidentally using machine learning in their day to day. Whereas now it is so, it is just so pervasive in everything that you do, where I just kind of wonder, there's just like a general malaise or a general comfort around like, okay, I'm already familiar with what a lot of this is. So, you know, what does that, what does that mean for me and the product that I'm trying to buy? Like, ah, maybe I'm not pushing back as much as I should be. That's a really interesting point, which is that if the people making the purchasing decisions or running these programs are already familiar with chatbots, they've got a general familiarity with what they can and can't do, right? They've got a feeling for it, right? Yeah, and I find that to be the most interesting because it's like, ah, yeah, but meanwhile, us cybersecurity experts are often on the sidelines watching people use these tools external to cybersecurity, being like, wait, wait, wait, wait, don't put your medical records on here. Be careful what you hook your machine up to and allow it to do. And then on the same note, we're building tooling that's being like, yeah, you could probably take your hands off the wheel. It's fine. We'll remediate things. We'll catch things. And it just seems to be the message just doesn't seem to be, I don't know if it doesn't resonate or if we're just not thinking through the potential impacts. But I feel like there's an opportunity for the entire industry to kind of take a step back and be like, what are we actually trying to sell here? And what does that look like? And that's kind of what I've been trying to communicate internally with this idea of like SAE levels for autonomy, for lack of a better way of putting it. But I mean, just going back to what we were talking about earlier, which is like, how do people gain trust in these systems? I mean, it sounds like what you're saying is the starting point, the starting level of trust is already kind of high because people are familiar with this, you know, with, you know, basic, you know, chatbot technology. So there's already that starting level of familiarity. And then they're working through this stuff mostly with questions. What are the questions that people seem most concerned with? I mean, you mentioned red teaming as being a big, big concern. Like, what are some of the other ones where people are really like, you know, this is a deal breaker question for us when we're looking at evaluating an AI-enabled security technology? I think one of the bigger ones I hear is just about data flow kind of through these agents, right? There's a lot of, I think, misunderstanding about what tends to happen. And I feel like the main fud around AI use in general is like, oh, these frontier providers are going to take all your data and train on it. It's funny how that became just an accepted truth when it is just not the case at all. That's not how this works. Right, right. And that's, you know, I do elements of that happen probably on some level. but cybersecurity industry in general has so many policies and guidelines that we need to adhere to with regards to data. It's like, we're not just vacuuming all this stuff up and then shipping it off to a frontier provider and being like, give me a response back, charge me money and keep the data. It's yours. That isn't really how it works. So, you know, part of it is education, right? with these customers and be like, look, this is the way this flows. This is what these models actually do. When we say we're learning from your feedback or from mistakes, this is what we mean. We're not going back as well. What's interesting here is that you're talking about where there's pushback. It seems to be, well, is it safe? They're like, is it red teams? Is our data contractually protected against being included in a training set? So it seems like people are not so much pushing back on can this thing do what it says it does they're more pushing back on is it safe to use it do you think that's the the dynamic here i think that's the start right so that's i can like pitch this more is those are the questions you get pre poc or pov and then once it's in their environment that's when that's when you get more of the operational questions hey what was this trained on, do these things look like my environment? If they don't, is it an approximation? If it's not an approximation, how do you learn? At what point should I feel comfortable hitting the toggle button saying, I need to be in an active kind of feedback loop as opposed to a passive feedback loop? And that is a really interesting thing to navigate because it really can be a choose your own adventure. What level are you comfortable with? And you can chip away at that as a vendor by giving them explainable kind of transparent reasoning along with any decision that it makes. Or you could just say like, look, you could treat this as any other feature. It's just slightly any other machine learning feature, but it's just slightly more intelligent. And we've found personally that it's a, it's kind of a back and forth, a give and take where you're showing them evidence, trust is built up. You make a mistake, trust can degrade, but then how quickly do you turn that around? Or is the explanation around why that mistake occurred strong enough where that sort of trust did not evaporate or go down? You just said something fascinating there, which made me, yeah, change my thinking, I guess, about how all of this works, right? Which is, you said, oh, it's just like a machine learning thing, but it's smarter. And in many ways that's true, right? Because LLMs are just machine learning, but like at ridiculous kind of scale that was thought to be like impossible previously. You know, you could do that, but you'd need so much compute. It's ridiculous. It's like, yeah, here we are like hundreds of billions of compute later. And that's what we got. No one expects machine learning solutions, which are just bought and sold like without any question. No one expects them to be perfect and never to make mistakes. But it seems like when it comes to a lot of these like contemporary AI solutions, that expectation is very different. I'd never thought of that before. But if your machine learning based IDS or mail filtering thing makes a mistake, no one's even complaining at that point. They might grumble about it a bit if the mistake's really bad. But why is it that there's such a higher expectation that these AI solutions are going to have to be perfect? People will point to them making a mistake and say, see, this technology's rubbish. They don't do that without tech. Like, why is that? Yeah. I chalk it up to, you know, the hype, like the marketing hype around AI in general is, is it like a, it's such a level that I feel like it's very hard to walk back. And, you know, I recall the days where it was like, our machine learning catches 99.999%. It's like, that's probably not true. but I think now there's just this expectation that even when you make a mistake, these things are so smart that it's just going to pick it up the next time around. It's like when you're, you're talking to a frontier model via chat and you're like, no, no, no, that's, that's a mistake. And it's like, it takes on the persona of a human being. And you're like, oh yeah, that's actually, that's a sharp question, or that's a good point. And I think human beings take that feedback and they're like oh it's it's learning so now now i shouldn't see that mistake ever again and i think where you run into problems particularly with being a security vendor is you're pulling in these frontier models you're not actively adapting them right like nobody's sitting 30 million dollars right yeah yeah yeah like you're saying oh people you know are they learning but they don't you know and even if you put you know uh like even if you prime them with the right instructions and prompts and whatever they still ignore you every now and then we saw this uh twitter thread recently where someone lost their entire production environment because they thought they're like text-based you know instructions to a model of never ever do this were guardrails and they're not yeah yeah and that's i i love that consumers are getting a little bit more savvy and and they're learning more of the nomenclature and and kind of what to ask so it is cool to get things like, you know, what guardrails do you have around that? And it's like, well, you know, here's kind of what we're doing and this is what we give it access to. And sometimes it satisfies things, but other times they pull the thread a little more and they're like, all right, well, talk to me about tool use. Like what tools do they have? And they're coming at it and it's getting, I want to say maybe more precise the way they're thinking about it. And they're starting to pull the right threads. So as you go through prop POC, they start asking you, well, how, you know, why do I need the AI? What does the AI actually do? Yeah, yeah, exactly. And it's just like, and then they get that, that taste, right? And they're like, oh, wow, this, this like takes care of this problem that I have, or I'm throwing too many people at. And you're like, great. And then it's usually at that point, they're like, could we, could we put it over here? And I'm like, well, it took us so long to get to this point. Like, let's take a breath. Let's learn. And then we can start to move it over. And it's, yeah, the parallels, I feel like with self-driving cars and kind of what we went through in the late 2010s is like certainly not lost on me, where it's just like, yeah, kind of funny, kind of funny. Bobby, we're going to have to wrap it up there. We are out of time. Great to chat to you about all of this. And yeah, for those interested, they can check out Sublime Security, a great email security product. Thanks again. No, thank you. Take care. that was sublime securities bobby feeler there big thanks to him for that big thanks to sublime for being a risky business sponsor and that is it for this week's show i do hope you enjoyed it i'll be back soon with more security news and analysis but until then i've been patrick gray thanks for listening We'll see you next week.