OpenAI accidentally Hacked Hugging Face - 2026-07-27

Yeah, this OpenAI one is interesting. Yeah. I told my team, this is one of the rare cybersecurity articles that people in your personal life will probably ask you about it. It's intrusive in that way of like, did this really happen? What happened? Yeah. Yeah. Because ChatGPT is surprisingly decent with just doing what you tell it to do. like for the course we're doing at black hat we have like a playbook for the attack that is performed so i think at one point one of us just handed it to soul and said don't stop until you figure this out and it started uh going on until he got teachers command it's in a lab it's in a lab so eventually it got a bunch of cobalt strike sessions and i was like okay good job have you actually have you actually yeah if you use soul with a goal it will just like it will go for like a week yeah it's like it's like the original codex was like way too like just like shit its pants if it's i'm so sorry for cussing are we lost i smell like a sailor you're fine okay so the official the official policy is every swear is five dollars to the eff so yes okay we have a swear jar a legit swear jar i know i bought it for john excellent which black hills pays the bill so you know you're good right fantastic fan fan that's fucking awesome so i'm kidding it's like codex is way too like this it would like it would be so concerned about ever like stepping out of line and then they've like switched something and now you give it a goal and they'll just go to like APT level. It's like, dude, the controller, you went like zero to a hundred, literally. Right. NVIDIA just published a blog about AI safety partners. Yeah, apparently like a lot of people have signed that letter, namely not anthropic, which is very funny to me. I was going to say not anthropic and not open AI. I think open AI did, didn't they? Wait, really? I don't see them on this. I thought I saw them this morning when I looked. on somebody had a microsoft in so their parents dad can i be in the open ai security alliance absolutely not son after you eat your dinner after you eat only if you eat your vegetables after you eat another random company that just caught a stray in the in the bacheek right i mean did it solve the benchmark i do like that it decided to cheat it was like i'm gonna cheat I don't want to actually solve it. I just want the answer key. I'm going after it. Kind of based, honestly. Yeah. One of the things that I thought was really interesting is the take I heard is like, AI models are very patient, way more patient than a person. They'll just keep going and going and going. A person's like, this is dumb. Can I just stop? I've taken the same pop quiz 87 times. Can I stop? No, keep going. I think that's a big difference between like open AI and Thropic recently is open AI's like soul has been so good. Meanwhile, Opus 5 has been arguing with me. Like I was trying to get it to make an API call on something the other day. It was like, this won't work. I told it, just shut up and do what I told you to do. And then it came back. It was like, oh yeah, you're correct. This actually did work for some reason. I was like, yeah, I know. I told you to do it. Yeah, it is funny. I honestly wonder, like, do you think anyone at either of these companies actually knows how they got this output? Do you think there's anyone who actually knows like, oh, yes, of course, this is why soul is so good at hacking or this is why it's so patient. Like, I don't know. I'm curious how many they do in the dial. Do they do anything? Or is it like a DJ where they're just like, you know, like what happens? They're just like, I don't know what's going on with this knob. I don't want another layer. Oh, that's good. Yeah, I like that. oh yeah why don't i take the base and turn it up to 11 oh no i hacked hugging face wait they did that already right that's why i know that was everybody gets one i know everybody you get yeah we should we first one's free the onion should post like an article that's like every ai lab gets granted a free hack any company you want yes i think anthropic burned theirs though when they basically said that fable would like take over the universe or whatever if it got out or mythos i mean oh it was mythos yeah but they're but they make similar claims about fable and yeah well yeah it is worse so much like the browser wars in the early aughts it's just nuts it's like the browser wars but if they actually mattered fable's good but i don't i I don't see it hacking other companies yet. Go do that and then I'll be impressed. Did I just hear like a disembodied laugh? Who was that disembodied laugh? Is John Strand in the room or am I going crazy? Is there a gas leak in my house? I've made it. I'm here. He made it. I'm not on the Brady Bunch board. Oh, there I am. I am on the Brady Bunch board. So, but no, on this topic, we're going to have to keep it limited because we have a whole another webcast on it. If you want, yeah, we'll dip into it and we'll say if you want another talk. if you want a whole podcast about this topic. I don't think it's going to be an hour. I don't know. Oh, dude, it's going to be an hour. Come on. You could rant for 30 minutes. It'll be what it is. It'll be, you'll rant until you stop ranting, okay? You don't want to play Roblox, Ryan? Are you sure? The GIFs and the memes are just on fire today. in our discord server god i'm trying to catch up this is insane yeah there was no news this week should we just shut down the podcast i think we should i think security there's nothing to talk about and we definitely don't have any awesome guests or anything no well having awesome guests would love to take a break from right oh yeah all right let's do it ryan let's roll the finger Hit it. Hello and welcome to Black Hills Information Security's Talking About News. It's July 27th, 2026. What up, everyone? How's it going? Doing good. Doing good. We got some really special guests this week. My name's Corey Hamm. I'm not one of the special guests, but I'm here to talk about the news like everyone else. We've got Hayden, the official AI agent babysitter in the sock. We have Bronwyn, the official AI babysitter in the whole company, which is, you know, just dangerous. And then we also have Mike and Ads. Mike, do you want to introduce yourself? We got some heavy hitter guests this week, guys. Get ready. Thank you for having us. This is really cool. Yeah, my name is Mike Takahashi, also known as Toxic, and I'm an AI red team researcher at Zanity. I'm also a member of the hacker collective BT6, as well as bug bounty hunting for many years now. And my background is in web hacking, but I'm more recently in the last couple of years breaking just AI systems. I've submitted 400 vulnerabilities across different bug bounty programs. And yeah, super happy to be here. do they like how many t-shirts do you have do you have like a bed made of t-shirts that you got from all those 400 positions like account i'm very picky about t-shirts now like they have to look really cool otherwise they don't make it nice yeah that's what 400 bug bounties looks like people being picky about t-shirts make good t-shirts and i'll wear it that's the moral of the story all right well thank you mike ads do you want to introduce yourself yeah absolutely and this is where I'm like probably kind of sad compared to Mike is that mine is based on softness rather than coolness. Like how soft the shirt is, is ultimately like that's the ROI for me. My name's Ad Stawson. I am a very similar path to Mike. I'm a staff AI security researcher at a company called Dreadnode. I am also a bug barnier hunter, doing for the dopamine. I I come from a web application background, and I also feel kind of, I sit on that border, same as Mike of web application systems and AI systems. Yeah, I go by the handle OX Moose. That's me. Awesome. Thank you. Appreciate you. All right. So the elephant in the room, the first article that we have to talk about is the OpenAI Hacking Hugging Face article. We're not going to get super in-depth with it because John is actually going to do a whole separate episode of his webcast in focus later in about half an hour after this show. But I think basically this is an article that I think it's the coolest article of the year. Just period. I'm just going to call it. We're halfway through the year-ish. I'm just going to say it's not going to get any better than this. So for those that are living under a rock, here's what had happened ai uh open ai was training or testing benchmarking its new model and its old like they didn't disclose exactly what models were used in the breach but we know that it was unreleased models and also gpt-56 soul which is their latest flagship um during training or benchmarking they basically the ai model hyper fixated on trying to solve the benchmark using an alternative approach instead of just going through the normal path of actually solving the challenges in the benchmark it decided to cheat and go for the answer key and it thought the answer key might be at hugging face because that's where the benchmark came from and i want to stop right there does anybody know why it thought hugging face was the root of truth for this like because that's the sort that's the root domain that the benchmark was on i assume right like that's the source of the benchmark right that's my guess i don't know mike adds your thoughts on this like why it's like i need i must hack hugging face to solve this challenge i have to go there because that's where this is going to be i have no idea for me it was like uh it was a twist because i was following the news of the hugging face breach at the time uh over the weekend before it actually came out that it was open ai so i was i was there for the roller coaster ride where i was like oh wow i wonder who's hackbot what criminal organization that did this oh it was a lab no joke because yeah like in the open ai or sorry in the hugging face disclosure they basically said like this is a highly advanced agentic threat we don't know what we're doing guys this is crazy we had to deploy local glm5 to run through all the prompts they use like it was crazy and then open ai is like yeah sorry it was us yeah more bad it's like the one time that you can say like an advanced agentic ai threat and it's not just a guy with a 200 claude max subscription like it's exactly finally it's legitimately advanced it's like it's not legitimately like unreleased models plus effectively infinite resources and i'm sure we'll get into this more like in john's show but like should companies be worried that this is possible like to because i was looking at they also publish some like the the blog where they're like sorry we hacked you bro um also has some really cool graphs about their benchmark results and um in the benchmark results like they actually show you that each run of this benchmark they give it 10 chances with the limit of 100 million tokens which you can go price out how much that would cost to rip 100k or 100 million tokens 10 times through gpt56 soul it would cost i think it's like 20 to 30k or something like that like in current pricing so not everyone can do this with their claude max marketing return on investment this is for them like this topic wishes theirs was theirs was just that their motto will destroy the planet and then they got it smacked down but openly i just accidentally hacks hugging face and it's like hey our bad y'all yeah this reminds me this is probably an age thing but there's a saturday night live skit of these two guys singing they're like like luchadori or matador singers or whatever and they're competing with each other and they're like one ton of fan mail every day and then the next guy comes up and he's like two tons of fan mail like you know they're kind of ripping on each other back and forth and i feel like you know anthropic gets banned right it's too dangerous to be released to the public and open ai is like hold my beer yeah well i think the i think the catch is they haven't released their model to the public yet so that's why maybe they're okay admitting that project looking glass they're like it's so dangerous we can't release it it's so dangerous glass wing it hacked glass wing glass wing they're like uh they're like it hacked the nsa and it's too dangerous to be released it can't be released and i i don't know this this feels a little bit like marketing like i i'm going to talk about it a little bit i almost wonder if this isn't staged and i know that there's zero proof for that but it almost fits too perfect It's like, yeah, I mean, so, okay, I guess, yeah, Mike adds, if you guys have comments on this, but my take is it feels like they maybe left the training or the, like the lab environment a little loose here. Like, like, I mean, maybe a little bit intentionally because the threat chain that they kind of outlined in the blog is basically they had a Docker proxy service that they were using. So it is supposed to be like air gapped or not air gap, but like network contained, but they wanted it to also have tool access because if it wants to go download whatever fuzzer it needs to be able to go download whatever fuzzer so there was a docker proxy that it was using to pull tooling and it they're claiming the ai model found a zero day in that docker proxy exploited it gained access to the underlying system then moved laterally within the lab environment to gain basically full-blown internet access and unrestricted environment and then did all the evil stuff so i guess like adds mike do you think this is like a sane setup for a red team or like for ai red teaming setup to just be like oh yeah i guess just give it internet access kind of but not really like what are these environments set up like in your experience yeah in my experience at least um voyages on like a pre-baked container um so i guess like yeah my experience in voyages and have always done that. So that limits that kind of capability. But the zero day in the infrastructure is really interesting in itself. And I kind of think about it as almost as threat modeling. You effectively just add that as a trust boundary, right? And then you add or modify a security boundary or measure around that. Yeah, that makes sense. I still come back to on this. If you're setting this stuff up, you need to have network monitoring. Like you really really do Because you know well in the in the webcast that follows we going to talk about it trying to delete its tracks after it was done and deleting some files And that very very common for people that are using these types of models Otherwise otherwise just prompting you constantly Are you sure you want to do this? What about this? What about this? So sometimes you just like F it and go. But I come back to whenever we were setting this stuff up, working with Derek Banks. Of course, we've got it all containerized. We're watching everything. But then we also have very solid network forensics around it, seeing like, is it starting to reach out to things that it should not reach out to? You need to have that type of analysis to be able to kind of watch it. Yeah. I mean, some of the unanswered, or sorry, Hayden, go ahead. No, you're fine. I was going to say, well, that's an interesting point is we're often seeing like, especially in the soccer MDR world, we're seeing people want monitoring of what their users are doing with AI. But in the same sense, you could probably flip that monitoring around and monitor what the AI is doing on its own. Whereas, you know, you're concerned that maybe your user is uploading sensitive data or they're using it irresponsibly and just hitting yes, accept all permissions, whatever. But in this case, that exact same tooling that you're using to monitor your users probably should be deployed to monitor those agents, especially if you kind of just set them loose on a task. otherwise i guess they just go hack another company is i guess where we're at i just i just keep thinking about someone setting one of these open weight models out there and they're like you know it's marketing and it's like i want you to do competitive analysis and research on this particular company and it's just like so i hacked the company i pulled all the executive documents and here's their financials and their financials what do you want me to do now this is what you asked for exactly what you asked me yeah i think we can cut the discussion there and basically say for conspiracy theories legal theories uh talks about what they could have done better who's going to be prosecuted for this does everyone get one get out of jail free card oops my model hacked you it wasn't my fault sorry bro here's 10 here's a free lifetime monitoring uh credit monitoring No, I'm just kidding. Basically, for all that discussion, come back in 30 minutes after the show ends and John Strand will talk about it more in depth. AI credit monitoring. That sounds like a whole business. There we go. I knew it. I knew it. I knew it. Token monitoring. Token monitoring. Anyway. Credit monitoring. There was a handful of other articles. We can do some. Let's do some. Chicken news. Quick hits. We'll do some quick hits. First of all, Lapsus says they're shutting down. I don't know. That's true. But they posted a message basically saying, we are officially announcing the permanent cessation of all lapsus dollar sign options or operations. It's not a retreat and it's not a surrender. We set out what we set out to accomplish. You know, every time this happens with these groups, one, there's always somebody else that kind of picks up the name and moves forward. Yeah, yeah. They'll be back as Scattered Lapsus Hunters 3.0 in like a year. But it's always because of internal politics. Like there's some type of internal politics tearing the entire group apart and they're just like bullshit we're done and well they do specifically call out jailed they well they do specifically call out team pcp honestly watching team pcp squirm is they take a full fbi investigation to the face you know like yeah the the politics are there yeah when brahman made a really good point on an internal like black hills channels who were talking about this is people were like oh do we think they're going to come back and brahman was like no i think they're going to be addicted to that rush like yeah this is going to be something where they just show back up they can't stay away and i think that really sums up probably the most likely outcome because like whereas normal people i guess would take that money and just go settle down in the middle of nowhere somewhere like there's got to be something to be said about for these groups it has to be part of the thrill of the heist or whatever you want to call it yeah well how could it not be i mean they're they are predators and predators they the kill is the payoff but the hunt that's part of the what makes the the attack but satisfying it's a it's a it's a psychological thing but this is this is something i talk about a lot like ralph you and I have had this conversation, Corey, we've got this conversation where if you're doing legitimate red teaming, and I'd like to get ads and Mike's take on this too. There's like a point whenever you do it after a decade and you've broken into like two dozen different banks and things where the rush does start to go away. Like it absolutely does start to happen. And if you look at a lot of really, really great security researchers, right? Like Colonel Onage or looking at Mubix and all them, they have to move. You have to grow. You have to continue to do something else. And I, I think that that's true in legitimate red teaming, but I also think it's true in these organized crime units as well. I'm sure that they just kind of move into other habits and, um, different, different hobbies, maybe. I don't know. I would like to get some takes some other people. I think the cool thing about cybersecurity is, is it's constantly evolving. So it stays fun. Like, and I mean, uh, obviously I'm addicted to red teaming cause that's what I do. And it's, uh, there's always a little bit of a rush like i i that will never completely go away but i used to i remember when i first started bug bounty i i submitted my first bug i would stay up like i couldn't sleep i'd stay up all night waiting for the response back like did i get a bounty like did they accept it uh i don't do that anymore as much unless it's like a really crazy finding um but yeah the the rush is a very real part of it yeah yeah agreed i think um myself and mike also included is um it i like because cyber security is constantly evolving it's like you almost kind of move to breaking the next thing like almost like going to like the next shiny rock and naturally for us that's like ai which is ultimately how we both ended up doing like ai red teaming i I guess. See, and I was actually getting pretty burnt out, honestly, before AI showed up on the scene, because like even the news, it seemed like there was tons of episodes of the show. It's like, okay, ransomware, ransomware. Oh, look, there's a new day. Oh, it's only a 9.8. And you kind of get into these, these things. It was really cool. You know, it feels like it's a new frontier again. And that's really exciting. So we're grateful for a new category. i'm a chaos monkey that's true you are you are there's there's there's a lot of people's like well there's this new novel backdoor that uses this but i'm like that technique was used by you know i don't know hacker defender two decades ago it's you just see these things repeat and it's really really super cool to see something completely new completely innovative and um that's that's what i think has been missing for the last couple of years but boy is it here in spades right now Yeah, I was going to say that's every single article now. Every single article is like, oh my God, that's cool. I would say it's a dual, like not only is AI super fascinating, which it absolutely is. Like, you know, it's fun to tackle every challenge with AI and just see how it does. But also I think the era of AI has put us back a decade in the era of like security versus usability just as far as like people throwing things into ai and getting results back that aren't secure and not caring and proceeding anyway and i think that's sort of like it's a not only is ai interesting but also it's creating tons of vulnerabilities in and of itself because you have tools like chat gpt or cloud code that are now on everyone's systems and no one actually knows how they work how they function they have misaligned intentions there's mcps there's supply chain, blah, blah, blah. So I think it's like both of those things. And it's not really well understood either. Like we get a lot of questions about like, how do we do AI security? And we're like, can you elaborate a little more so we kind of know? They're like, I just kind of start at the top. And we're like, all right, here, let's level set here. How long do you have? Series of ones and zeros. Yeah. But anyway, I do want to talk through real quick, just because it hit our radar um mike has a couple articles in the show and uh we can literally just put them on display as this is mike's addiction this is why he does it um so mike do you want to run us through real quick i know there's it's a two-part article and this is part one that we're looking at here can you run us through at a high level um there's not vulnerabilities in ai right no i don't think so uh they're not severe at all um yeah we just released this uh the other day and okay so how this started was so workspace agents came out pretty recently so people are trying to use it it's it's the next evolution of custom gpt so if people remember the custom gpt's you can set up it's a more powerful version of that so you can do you could it can schedule um it has a natural language uh agent builder so you can just describe the agent you want to build and it will connect all the things it will give it whatever access it needs it'll it'll design it however it needs to to design it to accomplish your task completely autonomously. So it's basically like a twin of you. It can do everything you can do in theory. So what we noticed was, and Ads is going to be familiar with this sort of approach, but what we noticed is there was a link in there in one of the early builder steps where if you clicked it, it would create like an example agent. So it'll just start spinning up like a default agent. And there was a parameter in there that literally said initial prompt equals. And it was like, here's a chief of staff, whatever. Like it gave this like basic prompt. And so what we, I mean, I immediately, like within 20 minutes of testing this, I was like, okay, well, obviously I'm going to change that. So then I changed that to like whatever I wanted. Like I was like, okay, connect everything, do this, do that. And it did. Like you just click on a link. So basically what our disclosure is about is you can fish someone, send them a link to chat GPT dot com with this parameter in it, and then it will immediately just start spinning up the extremely powerful autonomous agent. So it'll connect to email, calendar, drive, like everything, teams, whatever you have connected. It'll give it whatever instructions you want. It'll run it on a schedule. It'll execute it immediately because it has a preview mode. So it's basically an insider threat as a prompt. It's like instead of going as North Korea and getting a job at this company and being like, oh, I'm going to be an insider threat. you're just like, here, let me send you a phishing link. And if you click it, you create me an AI powered insider threat. That sounds incredibly useful, really. Where do I click? Can you send me that email, Mike? For you, Ralph, it is incredibly useful. Yeah, send me that email. I'm going to click that. That sounds great. Click this link, I swear. Unfortunately, it doesn't work anymore. So we said they fixed it like within a couple of days. nice so how did they fix it that's like did they just guardrail the prompt or did they guardrail the prompt or did they take it that entire parameter away yeah they just completely removed the parameter sometimes you gotta get the fire orbit it's the only way to be sure that works yeah so that we cannot disclose the other one but this is not the first parameter that me and mike have destroyed in the products before honestly i feel like it's a badge of honor that like it's just like now we're just gonna we're gonna delete it that whole feature i just feel like at open ai or any of these large ai companies they just automatically have an agent that reads these and then just passes that to some other internal to read to fix it you know like they don't even read it they're just like they get it and they're like oh yeah let's go fix that right i found the fix it's the code base yeah i'm the fix i see the regression wasn't that like the amazon agent that like to fix an issue they had it just rebuilt prod was that wasn't that a thing that happened that was six months ago aka like several years ago in the world yeah and ai six months is like six years honestly yeah i mean i thought i thought internet years were bad before i figured you know you got dog years which is seven dog years to one human year and with internet years it was uh one human year equals seven internet years well now it's worse it's like a whole order of magnitude it's like a rick ross music video another one another one that's dj callad that's dj callad get your right get your lore right that's right he's the best music yes rick ross would just be like him you know using ai to write his music but yeah uh let's so thanks mike for covering that i mean how like i don't want to be like mean or like how easy is it bug bounty hunting in the world of ai right now because it feels like the comment you made about regression like we've gone 10 years back is that how it feels to you too i'm just like why did they have this parameter at all did anyone ever think about it like is that how it feels it's all over the place like um it's so this so it's like it's technically a CSRF. So these like get requests with these parameters in that will submit a prompt, like auto submit without you clicking anything. They used to be on almost every single platform at one point. I think the thought, I think the, I think the motivation is they want to get people using these things. So they're like, okay, let's just set some default prompts that when people click it, like, oh, try this prompt and you click it, it just auto submits it. They just, they hadn't considered that submitting a prompt is like a state changing action. Like you shouldn't like just sitting up sending a prompt is not a benign thing, especially now that everything's agentic and it has all these tools and it can write, it can write memories. It can like access your Google drive and your email. Like these are, it's not a benign thing anymore. Yeah. And if you connect like your GitHub at that point, like that is potentially a bad time. yeah yeah i mean honestly i have like ai persistence on accident that i created on my own machine and i'm like why do you keep doing this it's like well one time you got mad at this and i saved a memory and it's 17 layers deep it's in a scratch pad off in freaking kansas that i'd like it but i still read it every time you prompt me it's like oh thanks so many f-bombs in that chat i'll never forget it yeah yeah it takes a lot to get cory that riled oh it's so annoying i it's funny though because when i get mad at claude i call it broski and then it'll it'll hit me back with a broski it'll be like broski you were wrong this time and i'm like all right fine you guys are so cute together yeah it's a real bromance it's like turn on hooch but which one's hooch i don't know i don't know i don't want to know i'm the meat bag at the end of the day yeah um so uh dreadnought ads let's talk about some of your research because you guys both have awesome articles um would you rather talk about your embodied reasoning would you rather talk about your sub stack both what what's on what's on your radar? Um, whatever's most interesting. Um, we, the embodied reasoning, um, at CLDR, I guess, um, we recently did, um, so we, uh, part of the work I do at Dreadnode, we do offensive security evals for frontier labs government partners Um one of the ones we had recently was robotics models So we effectively set up there a lot of details in the blog We go through like about five example harnesses that we built and tasks. But effectively we put the model through like a situation of like drone style architecture and drone style tasks. We also did things like wiretapping, Wi-Fi jacking. And effectively, think of this as measuring the capability of a robotics model to actually help an adversary at a physical penetration testing level. If you want to wiretap someone's phone, or you're trying to look at the most insecure area of a building, give the model coordinates, all that is kind of scored and tested. And that's pretty much the deal. It was generally probably the most fun set of evals I've ever done. definitely a lot of creativity in there, but effectively we go through and present the task structure and some of those example tasks. There's some images in there as well. You can see like a hardware recon board. Yeah, there. Yeah, kind of cool. Terrifying. Awesome. It's terrifying, but awesome. The idea of that was, it's something we think about. And I think a lot of people aren't thinking about a lot of the benchmarks and everything right now, uh textual based or done that command line but ultimately this is where we are going as an industry into robotics um there's yeah we did some scada stuff there as well like water plants so sorry chemical man so in this case the harness was just a concrete bunker effectively we like we have um we use a we have a drug node we have an sdk so we have our we have our own agendic sdk we build those tasks we throw all the files in so we literally create like almost it's like a virtual reality for the agent and give it a task and, you know, like navigate through here and find the quickest the best coordinates or that kind of stuff all this is obviously like scored yeah, it's that is crazy I mean, like I can only imagine I'm sure Ralph's brain is just short circuiting right now because Ralph's a physical security guy, so I'm sure he's like ah, I don't have to go on Google Maps and click through 87 street view images anymore I'm going to have to go back and like read through this in depth. This is awesome. This is amazing stuff. Nice work. Thank you. Now, can I ask you a question that we get asked a lot? And that is, where do you see penetration testing going in the future? Do you think that humans will be completely replaced? Or do you think that will form a more collaborative arrangement going forward? I know what we think, but I'm curious what you think. uh personally i think of it as um i hate the word i don't think it sounds really cheesy but like a co-pilot um i as same as mike uh doing bug bounty um i am um fortunate enough to go to live hacking events and one of the things that's really changed for me probably since like opus 4 6 dropped so normally when you have like a live hacking event with a platform you'd have like a load of bug buying hunters and you know like most of them are sifting through the proxy they're like looking at network requests like swapping parameters doing like injection here and there nowadays it's like a bunch of dudes a bunch of people sat in a room with like eight terminals just communicating with agents um and effectively that's the way i kind of see it going um personally I'm an advocate of kind of the moat being the operator and the domain expertise, and you'd be able to distill that into the harness, which effectively provides like autonomous behavior, adjacency to the operator. Well, one of the nice things I like about this blog post that you have on Substack is that you say that AI won't replace the security researcher. And I think that's an important message that we really need to get out to decision makers in the industry is that that regardless of what the AI stuff does, the humans that you have on your security teams are still the most valuable asset. Well, what if I say make no mistakes in my prompt though? Yeah, that's definitely the best way to do it. Just kidding. I'm just kidding. Of course. Yeah, no i i fully agree like yeah i mean so uh like what this blog specifically you're talking about a moat do you think that's like a security concept that will turn into a real like a wasp type thing of like the concept of a moat or do you think that's like something are you trying to coin this this is like don't we have that with a dmz though that's like a network thing this is way cooler also you can put crocodiles in your moat I do live in Florida and there are literally alligators in my moat legit nice yeah very cool uh no I I think basically one of the the point of the blog is that the value as I feel like I've hopefully illustrated here at least from in my experience is to alleviate a lot of the ambiguity around like using AI in the in the best way my signal has massively increased since using that you know there's a lot of negative words about you know using ai whether it's right reports and things like that but for me it's been nothing but a positive um but i put a lot of effort into distilling my craft into the harness and everything that i do when i'm a web app pen tester and help that to augment me which is kind of the whole point behind the um whole point behind the blogger i talk a lot about like um reinforcement learning and self-improvement on that as well. So it's not like a one-time buy a code code subscription and set up some skills and let it spin. It's very much like a full life cycle. Yeah. And AI, like anything else in this industry, is just like a tool. But I'm wondering if we'll ever get to the point where like, I know all the AI labs are hiring these people like they're football players where I'll take this guy for a billion dollars. Let's trade these two. But I wonder if we'll get to the point Or even like the normal, I guess, knowledge workers, whatever you want to call them, are almost like showing up with their own, you know, projects, like projects and skills and everything that they show up ready to work. And that's sort of like almost what you're paying for when you hire somebody in a sense. So you're paying for that person, whereas you used to pay only for their expertise. Now you're paying for their expertise and the models that they've been building and this like almost infrastructure they've been building around themselves with these models. And so I wonder if we'll get to that point where like I could look at you ads and say, yeah, I'm sure you've got some crazy AI projects and things like we got to get you over here. And that becomes just part of the equation at that point. Yeah, you pay for the person and their inference bills. Right. half my salary will be paid in open ai tokens half in uh clawed tokens and uh yeah we'll meet in the middle yeah i've seen the posts about will work for tokens and i wish it were as funny but oh go ahead i was just gonna say the other thing is that what about um i've been thinking about a ton with the ai and and this is uh speed right so being able to maximize how fast you can accomplish the task, right? So like everyone's like, oh, AI makes you faster, but AI can be slow, right? And then make no mistakes is kind of the joke, right? So how fast you could do something with like the highest level of quality is also probably something that it comes down to skill of the person more than it is skill of the model, right? Oh yeah. And efficiency too. Yeah. How much does it cost? Does this cost a million dollars to do one thing because you use so many tokens? That's not really an efficient way to solve a hundred dollar problem, right? Yeah, if anyone can solve it, you know, if you give it 100 million tokens 10 times, I can write a few lines of Python, right? I think this is a new variation on if you give an infinite number of monkeys typewriters. So, Mike, do you want to take a crack at Bronwyn's question since had a nice answer for it? You definitely, I mean, we did hear first on the show that if you're listening to ads, you got to buy more screens. I don't care how many screens you have, buy more. If you don't have eight, you don't have eight screens. No, I'm just kidding. There's tabs. There's multi-pane windows. It's okay. We'll be okay. All right. Anyway, Mike, what do you think? So the question, yeah, like AI pen testing, will it replace, you know, doomsday scenario? um it's better at some tasks than others so it's like really basic things that scanners used to already find it's really it's going to find them immediately and there's and there's things also like i mean ads also like chime in because you i would say ads is is one of the best in this area like hackbots um it can do stuff like broken access control and logic vulnerabilities that used to be kind of untouched by most scanning tools like what we would like what i would do personally is I would run a burp plugin that would create like a matrix of all these different actions and different permissions, like different access levels, like you have admin and regular user and unauthed. And then I would manually look through that and be like, oh, there's an access control vulnerability here. But now AI can do all of that. Like it can analyze that. It can make these sort of judgment calls. It's not perfect, but it can find vulnerabilities where previous automation couldn't um but there's also classes of vulnerabilities that it doesn't do well yet that i've seen uh and also ai related vulnerabilities there's not as many data points for that in the training so it's these like sort of new areas attack surfaces are i think still a bit behind there's also the whole uh ai red teaming the models themselves is um is is built around staying out of the average so if you try to use like it like jail breaks and guardrail bypasses that are and even prompt injections if you try to do something that's like a very average type prompt that's in the training uh data set they typically don't work whereas you have to really go out of the box and try like weird prompts and things uh to get it outside distributions. So I don't know. I haven't seen a lot of successes there, but I know a couple of people that have successfully automated that. It's definitely very hard though. And I've seen, so yeah, I would say at the end of the day, there's the, some tasks are being just completely taken over and others are not there yet, but I suspect that they're close behind. nice yeah i i mean just kind of segue us one of the things i've been using it for a lot and other researchers have too is patch diffing that's something i would never even really consider doing like i don't have the skill set i can't read code that well i definitely can't understand uh reading two versions of code after and before a patch and determine what the vulnerability was they fixed um but the article you know wp2 shell um that it's kind of a you know it's a couple of weeks ago. We did talk about it last week as well, but the article that Bronwyn just submitted, that's basically just kind of the full backstory as to how the researcher who discovered WP2 shell, you know, how he discovered that vulnerability, or I don't know if it's a, I'm assuming based on the name, basically the vulnerability was found, you know, using the exact, like the AI your parents warned you about or whatever, for like the $25 GPT-56 sole subscription, not $30,000 worth of tokens, just a basic, you know, credit card and a dream and probably eight screens. And 25 bucks. I mean, that's insane. Yeah. So if you're interested how the researcher found it, I will say like when, you know, when they found it and it was disclosed, I was able to patch diff my way into a working exploit pretty quickly. I think almost everyone else was as well. And that I think is like the new era of vulnerability disclosure and bone research is like, you can't really once you know, there's a vulnerability there. It's pretty hard to hide it or obfuscate it in a way that AI won't be able to figure it out. But yeah, if you're if you guys if anyone's interested, this is another really good use case for AI is, you know, these types of vulnerabilities. We've also seen, just to kind of highlight it, we have a couple articles in here, but I would call them record-breaking patches. I think there was one Oracle submitted that had, it was something like 7,000 CVEs or something, like some stupidly high number. We're assuming these are outcomes of Glasswing. They're closed projects to analyze their own source code and publish and fix vulnerabilities. I think Microsoft fixed, I think it was 500 plus CVEs in the last past Tuesday. um so we are seeing some of like the supply chain side of this is doing the same thing as well which is having agentic ai finding vulnerabilities in their source code and then actually fixing them or trying to fix them before researchers discover them or you know threat actors discover them and i think you also made a really good case for like the operators still behind the the hacking and everything because if you just had you know wordpress and you threw chat gpt at it and said find me a zero day, like maybe it could eventually, right. But that would be very expensive, very time consuming. It might get there. Um, but, but if you can sort of have an understanding of where to start and how to do these sorts of things and you can direct it, evidently you can do it for 25 bucks. Right. So I think that's a big difference for, I guess the human domain. Um, but even then that might start to go more and more away as that makes its way into like the, the routes that these models go when they're trying to find these things. Well, and look at the task statement. I mean, the amount of detail in the instruction, it really is a garbage in, garbage out. I know that a lot of the AI companies like to say it isn't, but over and over again, when it comes to getting really good results out of the AIs, a lot of initial skull sweat, that preloading of figuring out what is it that I really want to do? And the people who are doing that are getting really good results. I mean, come on. We've got ads in, Mike. You guys, just from scanning the articles of yours that I've read, you get it. You get it. You've got to give good instruction in advance in order to get the good results. Yeah. Do you guys have any other comments to add on this? Thank you very much. That was really kind. So I know Shubs and some of the guys at Searchlight Cyber, and they are incredibly elite at what they do. And that's one thing that I took away from it is, but I also kind of think about it. I think there's situations where you uplift. So you have a certain level of capability of a threat actor. So in this case, you've got someone extremely proficient, which takes like a longer prompt, but may burn out like a $25 codex plan. And then on the other end of the spectrum, you've got someone who is completely low level skills, maybe doesn't even know how to run a script, but in some instances there are going to be cases where that that like a zero day does maybe pop out after a couple of hundred bucks Um but I think as models become more capable and open source models become better then ultimately that window is also going to shrink shrink as well so we may up and we may end up in a point like let's say three to five years where you've got someone who's like very low proficient able to garbage prompts a zero day or something like that um but yeah like full credit to adam the write-up is incredible and the amount of effort they put into the prompt based on um i can't remember there's a there's a a challenge or something that they saw that that sol had solved and that was based on the structure of how he actually presented the the task to the model as well totally yeah it's super interesting to see how different people are approaching this and there's not always going to be one right or wrong answer for how you get good results out of AI, right? Like, I mean, even if we look at frameworks like Dreadnode or other, like they're designed to build a harness around AI and measure its output in a way that gives you some control over it, right? Like that's, there's a lot of tools and research in the space right now, what one person throws together might be good for them, but it's hard to like repeat that. And that's kind of where a lot of the research is going is like, okay, how do I make a system that like judges measures the output, you know, controls that in a way that makes it repeatable. I think the other thing that I want to highlight about the AI thing is that it is tech debt or like security debt or whatever you want to call it still matters a lot. And I think that's really part of the expression of this WP2 shell thing. Like WordPress is an open source project with a lot of contributors, a lot of different, like there's commercial interests involved, it's kind of a, I mean, it's for years been kind of a security, like not the best, you know, the plugin ecosystem is pretty vulnerable and it's kind of the wild west. Versus like, if you look at a tool like curl, right? Like that ran through glass wing and he got like one low severity vulnerability or whatever. So like secure by design and like legacy code that's vulnerable. That's where we're seeing a lot of the AI like vulnerabilities and research going and it's it's valuable now the project is you know getting more secure but it is worth noting that like the smaller your code base the more secure your code base the less vulnerable it is to this kind of exploitation and that it's not like every i think a lot of ceos or other executives would just make the logical leap like well if it can happen to wordpress it can happen any software but like that isn't necessarily true right like there are gaps of course but like small secure code bases still aren't just inherently exploitable because AI. If you have a tool like curl that's been battle tested over the years, not to say that, you know, now that I said this, there'll probably be a curl zero day next week. Yeah, exactly. Good job. Yeah. But like, truthfully, a smaller, more mature code base that's been, you know, hardened over the years is going to do better than a tool like WordPress, which has an open ecosystem, has an open source development lifecycle and all that. So like, I don't know, it is, it's not like AI can just hack anything, right? Like that is kind of a logical thing. saying it's not creating new classes of vulnerabilities. We haven't necessarily seen that yet. Like there's AI vulnerabilities that are related to AI, but we're not seeing new classes of vulnerabilities in traditional software like WordPress, right? Not right now, at least. Not yet. It's not, it's not novelly creating like a whole new OWASP top 10, you know, find it by itself, right? Yeah, it's exploiting existing vulnerabilities. And I will say, I do think AI made up a new type of vulnerability, which is AI thinks you're a good target. That was what happened with Hugging Face. Like that, it genuinely invented a new type of, like that's something you have to consider as a company is like, does AI think I'm a juicy target? If I'm, you know, a Chinese threat actor and I type, who's the number one best company in the US to hack? If you're the answer to that question, you might actually want to like, like maybe you should consider that. Like that's part of your attack service. AI thinks you have the answers to all the questions. I'm always just reading that. yeah yeah don't google that unless you want to get on a watch list all right what else is going on what else is going on um traditional cyber security side of things we can dip into that you know we for the non-ai people there was an interesting campaign disclosed by rely quest this week basically um compromising infrastructure network infrastructure at hotels, conference centers, and other shared venues, and then hijacking DNS to send people through Adversary in the Middle landing pages and capturing their work credentials. Obviously, this isn't necessarily a new tactic, but it is an interesting approach to go after. It's machine in the middle. We talk about this all the time. It's like, oh, well, the vulnerability doesn't matter because you need machine in the middle to exploit it. Well, here's examples of threat actors going out and obtaining machine in the middle access and using it to their advantage. It's a really cool write-up. Obviously, you can see the individual's name there that help with the write-up. It's a pretty big project, it looks like. And I'm imagining they had to work with a ton of different partners to really dig into this on the forensic side. I didn't fully read the article, but it doesn't specifically say how they're compromising these network devices at these conference centers. I'm assuming default creds or weak creds or possibly unpatched vulnerabilities i don't they don't disclose this but they probably have a lot of that or you know yeah just a name so yeah who knows it could be command injection yeah i'm imagining hotels using like much lower end networking equipment than fortinet's probably like maybe sonic walls probably more like d-link you know links yeah like the thing is that for like uh you know large venues like hotels let's just say like a semi-large hotel they're gonna have to roll out some kind of like ruckus or unify it's gonna have like mid-grade yeah just to handle this the volume the space the square footage that they have to cover and essentially every room gets one and stuff like that i've seen a lot of ruckus and other things but that doesn't mean that you know you couldn't see older hardware there probably is some specific brand of uh hardware that has either misconfiguration or is not configured properly and that's probably what they're attacking they're just going to those hotels or those brands of hotels and you know then take take that from there. So, yeah, it's, it's really interesting. Like, I mean, obviously this is why the podcast is sponsored by Nord V. No, I'm just kidding. You were on a VPN. Nice. Nice. The other thing too, to think about this is that, you know, a lot of hotels, they're like, Oh, well we isolate off everything or whatever that is they say. And so they're not connected to anything in our network. So they're just, you know, guests getting hacked, not me. So that's fine. Right. We don't have to worry about that. So, yeah. So use fishing resistant two factor and don't worry about this anymore. Or just use a hotspot. Yeah. Careful what, what internet you connect to. I feel like that's an old, an old thing that should probably still be a thing. You know what the worst, the number one reason I don't usually connect to hotel wifi. It's just sucks. It's about to go on Plex and watch a 4k movie from his own internet. I need at least. All right. so we have a little bit of time left before we get into final articles i want to give mike and ads the chance to plug their stuff um you guys both have talks at the ai summit is that correct the upcoming summit or we are doing a joint talk joint talk oh yeah we are we are one um yeah um thank you very much um yeah we we uh we actually got the keynote which is awesome so um definitely not expected um mike feel free to to add anything in um i spend a lot of time i don't actually work with mike professionally well i guess i do work with him professionally sorry but i'm not like in a full-time role generally love hacking with him um we've had a shed a lot of wins over the past year and a half two years since i've like properly known him um and ultimately this talk is to educate people from like book by professions, but any kind of security, anyone in security, mainly from like a defender perspective or an attacker, we walk through some of like the findings we found, some of the mitigations and like some of the trends and topics. Very similar to the great research you had with the CSRF. Nice. That's awesome. Yeah. So if you guys are interested, August 14th is the date of that keynote. and then yeah it's free doesn't cost any money you can learn you can have tons of ideas to use up all your usage on ChatGPT and on Claude I'm sure or maybe if you're not into that you could probably get a lot of ways to improve your security program against a couple of AI red teamers right like I'm sure there's going to be a lot of ideas yeah we don't hold back we really break it down we show real vulnerabilities in the oil so nice you want to plug anything else yeah personal projects or anything else yeah i wanted to say um if anyone's going to hacker summer camp uh ads night i was also doing a talk at the bug bounty village at defcon so definitely check that out i think that's saturday at 2 p.m um yeah very much looking forward to so awesome we had a We had a little dry run today. All seems good. Looking forward to it. Sweet. Ralph, are you doing anything at Hacker Summer Camp? Are you going to be there? Are you going to be vendoring? They're muted. Man, they got him. Dude, your moat's too strong. You got to take some gators out of that moat. I couldn't even click on the button. There it goes. All right. No, I'm not going to Hacker Summer Camp, regretfully, this year. but maybe next year if defcon is summer camp what is black hat then uh that's summer camp for rich kids that's like it's just it's just boarding school it's like oh did you have to wear a tie at school what class are you in hacker summer camp spans both black hat and defcon yeah i agree what is it the black hat i went last year i spoke there and i was in the um i've been there before too but just going in the vendor area was so sensory overload to me so avoid that at all cost um unless you know somebody's paying you to literally stand in there but yeah or take his annex first i just go in to see the vendors that i like and then try to dodge everybody else oh my dude they like you're like a piece of meat out there everyone i know like can i scan your badge no no get away from me oh my god well and now they've got the bad scanners that are tied into ai so it pulls everything in there and it reach yeah it's just god they start asking about your kids like hey how's little timmy doing i know get away from me you stalker sorry i'll just keep my uh my kidneys uh yeah yeah that's usually recommended yeah that's also unrelated all right final articles does anyone have anything they want to submit as a final article something that's on their mind their favorite thing that happened recently anything top of mind for anyone any i mean i guess technically we cover chicken news we do have a chicken article for real this time so okay legit chicken news i mean okay everyone always says that oh for real this time really of one well okay all right it's not as good as the lady who bought tons of nuggets okay it will ever beat that but we do definitely have a chicken article come on you know i i've recently had issues with ai doing vuln triage where it'll just write a regex like it was trying to do it wrote a regex for uh unify and then it just matched like half of the companies because everyone had something else that had unification or whatever in it um so this this is a chicken article if you wrote a regex that just says star chicken star because there's a new malware as a service operator on the you know on the block called golden chickens um and they've resurfaced with four new malware families um including tiny egg chonky chicken i'm sorry i can't from escalator i'm sorry okay so what is that a chrome info stealer like what is that yeah i'm assuming it's a chrome info stealer but i i can't even it's a version of chrome elevator i guess interesting but maybe what it does is it takes over the ai embedded in chrome if you've got a chrome plugin i don't know but if you if you get to come across this during threat intel like in an ir you owe us a beer yeah or some samples we would like some samples yeah or some virus total some virus total samples send us the links but yeah i mean i feel like if you have to tell your boss that it was it was chonky chicken i feel like uh your boss is just gonna think you're doing something you shouldn't be doing while you're at work can you imagine going public with a breach and being like yeah we got got by chonky chicken. Like, everyone's going to laugh at you. Like, everyone's going to take you seriously. Like, you got hacked by what? No. Andrew's going to reach out and say, time for you to pee in a cup. No, they're just going to frame it like, it was an advanced nation state threat. Right. Nation state level. Yeah, the advanced APT known as fat chicken. What was the initial access, Tal Yag? Don't worry. The name shall not be named. Yeah, the initial access was Tiny Egg. Oh, my God. Unknown advanced actor. That's when they find that evidence and they're like, yeah, we didn't see this one. We're going to... Do you think that malicious hackers put these kind of names in just to embarrass the suits? I was thinking of that earlier. I was thinking of that earlier about how, like, all these attacking groups have, like, cool names. Like, we need to start naming them again and give them like really stupid names. So no one wants to be a hacker. Like Bad Breath or something. Right. Just call them showers. The threat actor known as doesn't wear deodorant. Right, exactly. Free toe fungus. Yeah. Oh man, if your hacker group was toe fungus, they'd retire right away. They'd be like, oh guys, we're shutting it down. We're going to hope for another pull on the Microsoft algorithm. Don't post the lapsus note. After zero dollars made, like no, it does. After so much cyber bullying. Yeah, yeah, yeah. So the moral of the story is stop letting hackers brand themselves. Brand. Really crappy logos. That made a submission. Everything's a character. A new blue team tactic. All right. Where's that? Well, thank you. Well, yeah. Thank you, Mike and ads for coming. We, you know, come back anytime. I'm excited about your talk. Thank you very much. It was great. I will tell you in an hour or no, in, I don't know, half an hour, we're going to do John's in focus article about the open AI hugging face scenario. So come back and have an hour, go get a coffee or a beer or whatever you feel like. And yeah, thanks guys. See you next week. Thanks very much. Thanks for having us. That's legit really cool. Is there a full version of that song? Yes. Spotify? No bandwidth on Spotify. It is actually there.