We were all only supposed to care about the very top scores on AI leaderboards and the best vibes in our own workflows. But Three Frontier Labs just asked us, what if we can give you almost as good at a fraction of the price? The answer to that question might just be why the lead for OpenAI's super app just said in reply to an Anthropik post, which was giving away more usage, I smell fear. But more broadly, this video will be about giving you a dozen or so hidden gems strewn across the model releases, demos, background articles to help us all make sense of what happened in this last frantic, I would say, 24 hours. And that is all without even mentioning the somewhat disquieting post and paper from Anthropic about AI consciousness covered in depth on my Patreon. So I've tested the new GPT 5.6 Soul, Terra, Luna hundreds of times, including on my own private benchmark, and read everything I can, of course, by hand. Wait, that makes no sense. You get what I mean, manually. So let's dive in. First things first, there are three new models from OpenAI, 5.6 Sol, Terra, and Luna. Sol is only available on the paid plans. This combines with goodness only knows how many effort levels. On first sight, it looks like five effort levels, but there's a hidden pro mode as well. But the good news is that the observations hold broadly across the board so don't worry too much about that because the truth is across most of the benchmarks whether you're comparing the biggest soul versus fable or terror versus opus or luna versus sonnet generally speaking the open ai model will cost about a third of the anthropic flawed series and nor by the way does that mean you're always getting worse performance for that much reduced cost okay i flagged up this benchmark agents last exam where as you can see, the top scoring GPT 5.6 Sol on Extra High scores almost 54% and that compares to Fable going all out on max getting 45%. Yeah, whatever, just another benchmark, Philip. But wait, this was co-led by UC Berkeley, covers 55 industries. 300 experts were involved in crafting long horizon tasks that were proven to be economically valuable. The legendary lead on the benchmark, Dawn Song, said every task is derived from a real project that a human expert previously completed. No vibes, no human judges, fully reproducible. Okay, fine, you might say, pretty impressive roster of people who oversaw the creation of the benchmark, but going from 45% to 54%, is it that cool? Well, aside from the cost reductions involved with Sol, I would also point out that we never had to pass 90% on frontier coding benchmarks for developers to stop manually coding. There wasn't a singular benchmark that we beat, I mean, where we switched from hand coding first to AI coding first. So what's to say that might not now happen with finance or many of the other industries covered by agents last exam? I heard a report just the other day on Bloomberg where Corey reported this month that they are backlogged by tens of billions of dollars in demand from financial firms alone. So might we reach AI first for finance and many other white Collar domains by the end of this year, where you first get a model to do the task on the swanky new work tab of ChatGPT before you review the output and tweak it. One benchmark some of you might be thinking could well have been gamed, so what? Well, Agent's Last Exam is a pretty new benchmark, so harder for its answers to be found contaminated in the training data of GPT 5.6. And so I might add is Automation Bench from Zapier. Feels like just a few weeks ago that I covered the release of this benchmark. Again, it tests AI agents on end-to-end workflow execution using real tools across real business functions. Sales, marketing, operations, support, finance, HR, built on real patterns from monthly tasks done across millions of companies. This time, the performance per dollar is not as stark a lead for OpenAI. You can see the 0.7% score lead for Sol on Max, that run costing almost the same as for Fable, but still. Similar result, by the way, in the previous most famous benchmark for measuring real world impact, GDPVAL. This time, Fable actually has a slightly higher ELO, albeit at triple the cost Couple more impressive examples and then the counter argument lest you think I biased towards OpenAI Artificial analysis combined multiple coding benchmarks into one aggregate analysis and you can see what it found. Lo and behold, GPC 5.6 Sol scoring the highest, getting 80 on the index versus Fable 77, again at a lower cost. Might seem like this is reinforced by the scores on Terminal Bench 2.1. Think of that as a model's ability to complete fairly complex tasks using the command line terminal like writing, debugging, running software, multi-step tool use. But wait, it must be added that that artificial analysis coding index covers the very same benchmarks. Terminal Bench, Deep Swee. What I'm trying to say is that this little collection of benchmarks might make it seem that Sol is better even than Fable on its favorite domain, coding. But it's two measures and there have been questions about DeepSwee and two more recent lauded and harder benchmarks, FrontierSwee and Swee Marathon, did not have GPT Sol results published. In the case of Swee Marathon, software engineering marathon, involving multi-hour tasks with tens of millions of tokens per trial, you'll notice Grok 4.5 in the lead. All that data that Grok now has from the cursor acquisition by SpaceX AI does seem to have really helped propel Grok. You'll see Fable 5 trailing on this benchmark. Also bear in mind this, which is the same argument that might tempt you to go from Fable 5 to GPT 5.6 Sol, the fact that it might be almost as good but a lot cheaper, might also nudge you toward Grok 4.5 or maybe the slightly cheaper still GLM 5.2, a Chinese model. When those models are added to the chart, OpenAI's curves might not look as appealing. Okay, but that point may have shrunk your enthusiasm a bit too much, because if we turn to an abstract pattern recognition benchmark, ARK-AGI 3, the successor to some of the most talked about abstract reasoning benchmarks in the industry, oh and graded by the way to be especially penalizing to models, GPT 5.6 Sol still does well. Yes, it gets just 8%, but compare that to other models struggling at below 2%. I think Anthropic didn't even run Fable because of the costs involved. Then there is competitive coding. Just in the last 24 hours, it was announced that an OpenAI model, possibly an internal model, literally broke a competitive coding benchmark, just aced it. Kind of a slight warning shot that if a domain is verifiable, If you can check an answer is correct, then before long, there will be a model that crushes it. All these other sub 100% scores that I'm spending most of the video talking about is more an artifact of those domains either having messy data that's hard to verify, or of there being just not enough of the relevant training data inside the models, or the models not being given enough of a reasoning budget. Which brings me to another benchmark I want to cover, maybe a whole new class of benchmarks, which is that companies are now making entire games, playable games, as part of their release notes to show off the capabilities of their models, demonstrating that they can create tasteful, ergonomic, and functional interfaces, showing off that models can use a browser to check the results of what they've created. Indeed, you can do the same. I ran the very same prompt that I used for Fable on, this would be 5.6 Sol Ultra, and out we got this game with a title page that I think is significantly better than Fable's output. I will say that 5.6 twice mucked up the sound settings though so it's not all smooth sailing. I've published the minigame by the way in the description so you can play if you like. My quick summary would be that it's not as visually stunning as what Fable came up with but I love that the little companion, this is essentially a Pokemon clone by the way but set in the Redwall universe, but you can actually see the companion So when you move, it just follows along. That's pretty cute. If you use the lightning setting, which does use up more credits, of course, I got results within 20 minutes where Fable took more than an hour. But here's where I want to bring in Meta's MuseSpark 1.1. Because one of the most prominent benchmarks that Meta celebrates is its ability to Vibecode. They cite Vibecodebench. This was created by the independent vowels and if we look there you can see MuseSpark getting a score that not that far off Sol 72 versus 81 but at around 35 times less cost This is that same awkward cost efficiency point from earlier. OpenAI can't lean too hard into their model being almost as good but cheaper if there are other models like those from Meta and XAI that are almost as good as the GPT series but way way more cheap. The reason I bring up vibe coding is that if you're into say game design mocking up a website more consumer or prosumer use cases then maybe you don't need sole offer all. Maybe you don't even need the much lighter Luna that's still on this benchmark four times more expensive and much slower. Meta's new MuseSpark doesn't seem that far behind in computer use either. We'd need to see a much wider range of benchmarks but the promise is there. It's early days of course and you might question how such a small and cheap model scores so highly on humanity's last exam. Remember, that exam was co-created by Scale AI, which was bought by Meta. Data contamination, anyone. But we shouldn't be too dismissive. There's GLM 5.2. There's a range of examples that near frontier performance can be had for much, much less. Oh, and I've got this far without even discussing my own private benchmark, SimpleBench. Recently cited in The Economist and MIT Tech Review have reached out for an interview. But essentially it's a trick question or common sense reasoning benchmark. An absolute veteran these days at over two years old. Anyway, OpenRouter exposes a somewhat unannounced pro version of the models. And with Sol Pro, we see 71.7%. Still is 10% lower than Fable though. And actually, not that far ahead of Grok 4.5. Big credit to XAI, Grok 4.5 seems to be a genuinely good model. For completion, Sol itself got around 65%. And I've also added Sonic 5, Kimi K 2.7, GLM 5.1 and 5.2. And more importantly, I've added a timeline so you can see how models have progressed. And a performance per dollar Pareto Frontier analysis chart. Essentially, what score do you get for your dollar? The standouts there for me would be QEN 3.7, GLM 5.2, and actually DeepSeek V4 Flash, an incredibly cheap model, but still gets almost 50%. Now, if you don't care as much about cost and you want great results fast, then the release video from OpenAI demoed the new Ultra mode, where essentially it's a bit like the DeepThink mode for Gemini, or indeed the Ultra mode for Claude. More parallel agents gets the job done faster. I mean, this is just one benchmark about export generation. Still, the trend should hold. I'll end the video on why I think we're not even close to saturation on these kind of approaches too. Then the point you probably expected me to spend more time on, the whole self-improvement argument. The fact that GPT 5.6 accelerates OpenAI. So goes the claim. The launch video added that Sol post-trained Luna. But these claims are very hard to judge. Did it fully post-train Luna? How much human review was involved? How much hand-holding to get it to the place where it could then post-train Luna? Did it do a worse job than the OpenAI researchers who attempted the same thing? After all, we know that even models like Fable 5 are nowhere close to saturation of post-training benchmarks, like Post-Train Bench. I can well believe that OpenAI are using pre-release models more and more and more, with output tokens doubling model to model, as they say, and more and more of their research budget dedicated to just using existing models to accelerate their own research. I do feel though that Anthropic were more honest with the caveats that come with this. They said in the Mythos system card that all of this quote productivity gain is about an order of magnitude short of the kind of gain that would be needed to just 2x their own research speed. So don't naively read this hundredfold increase in internal coding inference as being anywhere remotely close to a hundredfold speed up. Maybe research at OpenAI is going 20-30% faster than it was this time last year. Big maybe and there's many other variables like number of staff but I do give them credit for introducing a range of other self benchmarks that I will dive into in future videos They also gave us additional secret internal benchmarks like AGI Index V5 That's an internal OpenAI eval. Spans apparently work, coding, research, computer use, science and cybersecurity. The numbers won't mean much to us, but Sol gets a new high score. By the way, you might want to look deep into the notes at the end of the release page because there's some other bonus benchmarks like management consulting tasks. Again internal unreleased but Sol scores higher than any other model including Fable. Here though is one benchmark that OpenAI are probably not as proud and maybe even a bit worried about breaking which is that the UK AI Security Institute found that it was now easier than Fable to jailbreak not just narrowly but with a universal jailbreak. That's the key unlock that allows you to then use the model for a range of nefarious activities. Not just get it to say a single thing, but make it do long form, agentic task completion and exploit development. The Institute says that we found these jailbreaks within hours. Indeed, they even appeared to preserve the model's capabilities. So this wasn't at the sacrifice of performance. Those specific jailbreaks, the Institute says, OpenAI has been able to mitigate. However, we expect further red teaming to surface similar jailbreaks. It's lucky that Xander Davies doesn't work for Amazon. otherwise 5.6 Sol would already be shut down. Is this why an anthropic researcher directly commented on this development saying that the ease of jailbreaking combined with the high rates of reward hacking talking of GPT 5.6 has him pretty worried about the alignment of that model. He adds I hope OpenAI didn't rush this model release just to keep up with Fable. Will the US government intervene to block it? I'm going to end it here but I can't help but also celebrate the new real-time voice agent that I've been testing out from OpenAI. Honestly, I really do recommend checking it out. Yes, the live demo didn't go too well on the OpenAI live stream, but it's actually insane. It listens while it speaks. So interruptions are just a lot more natural. Quite hard to convey in a video, but I think real-time translation is a huge win for society. Massive credit to the OpenAI team behind real-time voice. An especially impressive week then across the board from OpenAI, Anthropic with their consciousness report, XAI with Grok 4.5, and indeed even Meta with MuseSpark. Here's the thing I want to end on though. With all of that said, we are not even close to the end of model improvement. It's an ultra cliche, but I think we're much closer to the start than the end. Here is just one quantitative reason why. Four years ago, next month gpt4 was trained august 2022 way before it was released and that model was just under two trillion parameters we don't officially know but the best guess as to the current model sizes are around four trillion parameters for gpt 5.6 sol and 10 trillion parameters for fable which is part of why it costs more but wait even if that's right for fable that's just five or six times the size of gpc4 why so little parameter progress when compute availability has more than 100x since far more even than that i think well of course because there have been other demands on that compute going from a few million users for example with open ai to a billion branching into other modalities image voice video burning through a lot of that compute with long tasks with looping agents on ultra mode here's the point i'm making though new hardware is coming that can directly unlock larger model sizes, but even if that hardware wasn't coming. Eventually, if token usage ever even thinks of plateauing, then all the new compute that's coming online could be used to serve larger models, say 100 trillion parameters, or there have been public comments about a one quadrillion parameter model coming one day. That would, by the way, far surpass the number of synapses in the brain, as loose as that analogy is. That's without even touching on all the other axes of improvement that are left. So an actually wild week in AI. Thank you for joining me to think about it. Let me know what I inevitably missed. Do check out the Patreon post if you're interested, but above all, have a wonderful day.
