Nobody Is Getting New Manager Training for Their AI Team - with Dan Klein, UC Berkeley

If you're working as a human with another human and you're trying to delegate to them, you do trust them to come back and say, well, I actually couldn't find this information for you or I got blocked as opposed to I couldn't find it, but here's my guess, a wild guess and I'm not going to tell you it's a wild guess. That would not be good behavior from a human, but we see it all the time for machines. The systems we've built, really they are fundamentally systems designed to produce outputs indistinguishable from the truth. That's different than outputting correct answers. They're fluent, they're confident, the parts we do understand look correct. We assume that everything else is correct. And that's not always true. Hi, I'm Dan Klein. I'm a professor of computer science at UC Berkeley and CTO at Scale Cognition. I'm excited to talk to you today about hallucinations and reliability in AI. Give us a sense for your background and why somebody who's listening to this episode would go, ooh, I need, this is one I can't miss. Like what, where are you coming from in the world? Well, I've been thinking about artificial intelligence for a long time and my background is in natural language processing and human language. And so I've been thinking a lot about how we can build these sorts of systems. And so much has changed in the time since I started, you know, my research work in, you know, when I was in grad school, the big problem in natural language processing was like finding the verb. Well, since then we've, like we found the verb and we've got other issues now. And a lot of the problems in artificial intelligence historically have come from systems working, you know, too poorly. Things not, the technology not working well enough. And a lot of the problems now are coming from this contrast between the ways in which it works very well, maybe even superhuman. And then of course, still the ways where there are gaps and it's those gaps that really are still a problem. So my personal interest right now is in trying to figure out how to make systems which are reliable and trustable. And that right now is a big cap. And so I presume you're kind of alluding to what's known as the jagged frontier where, you know, some of AI's capabilities dramatically outperform, others don't, or others dramatically underperform it's disappointing. And the fact that there's jaggedness causes perhaps some jadedness, I think. One of the things I would love to learn about if you have comments on it is, I've seen, or I've heard anecdotally at least, that more experienced individuals tend to be able to navigate the jaggedness of the jagged frontier. Do you find that to be true? What helps someone be a deft navigator of the weird, unpredictable capabilities of these models? Yeah, that's a great question, I think. And also, that also comes down to really important questions we have to face as a society about digital literacy. And the capabilities of the systems we're talking to are very different. So I think a good example of this would be something like search or machine translation. If you think about the technology in, say, the 2000s, when you would enter something into a system like Google Translate and you would get a bad translation out, it would also look kind of bumpy. And you could tell pretty quickly that the system is influent. Therefore, it's maybe not accurate. Or if you are, we're doing a search in the sort of standard way we do search, you type in your query and you get back results and you can see, well, some of these are relevant, some of these are not relevant. And you go into that search process knowing that you're going to have to be doing some filtering. Systems today hide a lot of that from you. The systems are very fluent even when they're wrong. And when systems things fluently wrong and you've built up all of these instincts that fluency correlates to accuracy, it's very easy to not notice mistakes. Define fluency there. Fluency here is really about the appearance of truth and the smoothness of the language. And the systems we've built, really, they are fundamentally systems designed to produce outputs indistinguishable from the truth. That's different than outputting correct answers. And that means that there are a couple of problems. One is even the system itself doesn't know when it is outputting a correct versus an incorrect answer, when it's guessing. And the reason for that is it's always guessing. It's just sometimes it guesses right. And that puts a big load that we're not used to as users of the systems basically confidently and fluently giving us answers, which are sometimes right and sometimes wrong. And you can't tell. Why the why the load? Because as you just said, I actually love the idea of framing this as digital literacy, because I don't think we've had a guest that really talks about it exactly like that, which is very cool. Now, you contrast it with Google and you described what I think is very familiar to most of us, where we get a bunch of results and then it's incumbent upon us to sort through them. Now, why is it any different with an AI? Is it because the appearance of confidence that lowers our own inhibition or low? Why is there a difference? I think it's two things coming together. I think it's partly how the technology works. So if you go to something like chat, GPT, fundamentally, all these technologies, anything that's backed by like an autoregressive next token predictor, the way they work is at their core, they're predicting the next token based on what's come before their completion engines. And so if you and it's kind of raw state, you have it complete the sentence, the population of Berkeley is what comes next? Well, the system, it's not a database. It's not like there's an entry or not, and it has a metacognitive awareness of whether it knows the answer. It's just a matter of what density is predicted over these next tokens. And some numbers will come out. And because the system has such a generalized knowledge of language and context and many aspects of how the world works, the population is going to be the kind of population a city would have. In fact, maybe it's seen enough web pages that it'll actually output the correct number. Or maybe it'll output a plausible but incorrect number. All you see is the population of Berkeley is and then a number. And you don't know whether it's right or not. There's no certificate of truth that comes with that. There's no process that the system went through to determine whether it did or did not have that knowledge in some discrete way. There's only a claim presented fluently and confidently. And that means it's the load is on you to figure out is this one of those times where it's fluent and correct? There's a sort of times where it's fluent and incorrect, as opposed to a lot of experience where you're like, all right, I'm going to click on this link, half of them aren't right. I'm going to look at it. I'm going to check for signs. This web page looks sketchy. Maybe it's not reliable. This translation's got a bunch of disfluencies. Maybe other things are going wrong. And all of those use that we've been trained to detect when the AI fails have been really taken away from us. So the combination of the underlying holes and the misalignment with our experience with past technology ends up being an issue. Maybe that's a good segue to your startup. I guess if you can call startups these days, I guess a lot of, I don't know the terms. But you're talking about not just minimizing, but completely eliminating them. So how, how can this technology do that? I think the best place to start that answer is to talk about why the current technologies have hallucinations, which really starts with what is even a hallucination. So we talked a little bit already about how a standard transfer model is basically designed to predict the next token and the next one and the next one and the next one. We like to talk about as humans systems this one, oh, that was true. That was correct. We can talk about, oh, that was a hallucination. What's that mean? Right. These are judgments that we apply that maybe don't fit the technology or we talk about the system deceiving you. What's what is what is a deception? These systems are fundamentally today. They're just probabilistic systems designed to output plausible continuations. They output plausible next tokens and they are in as many ways as possible going to have the trappings of truth. They're going to be linguistically fluent if, you know, two words are correlated. They're going to take those correlations and they're often going to be, you know, and for example, in a RAG system, they're going to be looking at a retrieved information piece that maybe it'll choose to keep intact. And so in practice, these systems often the tokens they output are correct and then they're not correct and you can't tell them apart. So we call it a hallucination because it's confidently wrong. But to the system, this is all just its natural operation. We could talk a little bit about deception and what that means. It's another topic. Well, it's kind of malicious, right? I mean, deception to me implies mal intent. Is that fair? Yeah. As humans, we talk about deception that involves an intent to deceive. But this really gets into the topic of alignment and it's very, very easy for systems to become what we as humans might call deceptive. Sometimes these labels fit and sometimes they don't. So for example, let's imagine that we're a shipping company and we want to build an agent that's going to answer questions about package status and you call up and you say, Hey, where's my package? Now, this system, there's a lot of ways you could build this sort of system today. One thing you might do is you might decide to take whatever system you've built and optimize it through reinforcement learning. When you train a system through reinforcement learning, you give it some metric and you say your whole job and your action selection is to do well on this metric. And so maybe you tell it, okay, your job is to get a high net promoter score from our customers, which makes sense. You're trying to make your customers happy and so you tell it to optimize. And the system over its operation comes to learn that people actually do not like being told that their package has been lost. And in fact, they much prefer to hear that it's arriving tomorrow. So you say, where's my package? And it's actually lost, but the system tells you, oh, no, it's actually going to be there tomorrow. And it's it's it's seeking the reward of a high NPS. It is doing exactly what you told it to do, which is to choose actions, which makes the customer happy. And so then you can get into this process of saying, oh, well, you know what, maybe I didn't mean make the customers happy at all costs. Maybe I meant, and now you're in this very, very hard problem of trying to specify exactly what the system should be optimizing, how it should trade off truth and happiness. And is a system that makes that error that says your package is coming when it's not. Is that a hallucination? You're reasonable to call it that. Is a deception? Well, it feels like it might be deceptive because in a human, that sort of action would be characterized as deception. What it really is, is just efficiently optimizing an objective, which maybe isn't what anybody really wanted. Is this why I think I read somewhere you said that people building agents on top on the foundational models is good if kind of fundamentally broken approach? Is this the core reason? Yeah, there's a lot. There's a lot right now. There's a big activity out there and in industry of taking these sorts of foundation models as they are today and building a thin wrapper on top of that to build an agent. And that hasn't really worked very well. And I think what you're pointing out is exactly right that these are the reasons. So you have these systems which are inherently non-deterministic. And if you call up and say, you know, maybe you're calling up your package agent or you're trying to get a refund or change your flight or something like that, you aren't really looking for a system that uses all of the incredible breadth and strength of today's frontier models. Like you don't want that it knows a lot about quantum physics and can give you your answer in iambic pentameter. What you want is you want it to truthfully and reliably reflect what is in the database. And when it says it did something, you want it to actually have done it. And those are the places where current systems are weak. And so you have this misalignment with these the systems that people are deploying are strong in ways that they don't even really want. And they're weak in ways they need. And this is fundamentally because they're building on and like it's a soft probabilistic technology. Typically, it is hard to build a reliable and deterministic technology out of non-deterministic elements. Is the is the hack there to kind of chain it together with deterministic rule based kinds of whether it's automations or triggers or things like that. Or is it there are more foundational work around that you're advocating or building? Well, the most common approach out there is exactly what you said. The most common approach is to chain things together. And if if you think about that, you have some system and it is going to do something noisy that's not reliable. You can then bring in a second large language bottle with instructions to check the first one. And as the joke goes, now you have two problems because you've got noisy systems checking noisy systems. You get a cascade or you run, you know, 15 of these in some kind of constellation and any of them can make mistakes that might get caught or might not. And you pay this high price and latency. You have to run a model to check a model to check your model, which then gets checked by some other model. It takes a long time. It burns a lot of tokens, which is great if you're in the business of trying to like use as much computation as possible. But if you want small efficient systems that get it right in the first place, this is not really going in a good direction. What we do at Scale Cognition is instead we build models whose fundamental operation is different and that come along with a big class of determinism that we can guarantee because of how the model is structured and how it operates. And you mentioned chaining together with the rules. Another approach that people use today is they take this LLM, which really like as an artifact does a thing we've built is incredible in its potential breadth. Like you can ask chat GPT about anything. I mean, the answer you get back will be confident and maybe wrong, but you can kind of ask it about anything. And, and now you have the system and it's hallucinating. It's maybe telling you it, you know, canceled your flight, but it didn't or the other way around. And you're trying to get this system to do something reliable. And the instinct people have and the only real tool they have is to just squeeze down its domain until it's doing almost nothing. Like the deterministic rules you're talking about, you say, all right, LLM with this incredible power, all you get to do is decide whether the user said that it wants to talk about payments or bill. And that's it. That's all you can do. And so you've got a system that's like being asked to do only this small thing that it's bad at. It's like an 18 wheeler to deliver one letter or having a Porsche, but you only you only push it down the road, you know, with your I'm trying to think. Yeah, it's like an 18 wheeler delivering one letter where you've been told the most important thing is you deliver it silently. And so you're like, all right, we have go really slowly. Maybe it'll be silently. And, and the thing is you're wasting the volume that this 18 wheeler has, you're wasting its power and there's a better solution out there, which is get somebody on a bike. It'll be silent. And what we're doing is building models which have completely different control and performance profiles that line up better with this industry need for determinism. Given the, because to me, it seems we should go into what are the strings and maybe what are the trade offs of the skilled cognition model just for the record or just to kind of help us put things in perspective. Given the flaws that you've described with LLMs and, you know, in this amazing horsepower, 18 wheeler, et cetera, what do you think they're good for? Do you do you think they're good for anything? And if so, what are the use cases that you go, of course, you should be using the horsepower here. Yeah, I think they're good for a lot. I mean, the key is really in the name. These are generative AI systems. They're good at generating, good at generating content. And people complain about hallucinations in these cases where reliability is important, accuracy is important. But in many cases where these systems are the strongest, where they were developed, where they took off first, the hallucination was the product. And so, for example, if you're mid-journey and somebody comes to you and says, hey, I'd like a picture of a mouse holding a balloon, right? Not only is the whole purpose to get something new and creative that matches what you've asked, it's actually very important that it not replicate something at scene. You don't want a copyrighted picture of Mickey Mouse, right? And so here, what we would call hallucination in another context, the kind of confident and fluent, which in this case has to do with visual fidelity and sort of plausibility of the elements of the image. Here, you want the fluency with the creativity. If you go to chat GPT and say, give me an idea for a short story, you don't want something that Isaac Asimov wrote, right? You want something that is new. And actually, there, part of the problem is you actually can't tell if it's new. It's new to you, but how do you even tell? And so in all of these cases, you're asking for the power of creativity, of generation, and you're asking almost to be guaranteed a hallucination. And then you take the same technology and you turn around and say, actually, all right, I changed my mind. Now I want you to only do accurate things. I want you to only reflect the state of the database and I want you to follow these rules precisely. And they're just not good at that. They're not built for that. I guess just to be a little bit nerdy, but not too nerdy because I'm not. The nerdy are the better. I read somewhere that you only use synthetic data. And as I understand it, AlphaGo is the only other kind of operation that do that. So I am curious about this idea of doing this only as a theory. Only the other thing that was interesting as we talk about how you build it is that as far as I can see it, you haven't raised that much money. And so you're building this very powerful model, it seems powerful less and like it can do what it's supposed to do model, but you're doing with a relatively small team. And so yeah, how exceptional team. How do you square this idea that there seem to be this kind of like completely weapons raised where everybody's like trying to hide as many people as possible, putting as much money as they can into these models. And here you're building this kind of what seemed to be quite unique model, but less money only uses synthetic data. You know, like not that many people. Could you just talk a little bit to that? Yeah, I'd be happy to. So let me talk about a couple of things because there's a bunch of interesting questions in there. One is the question of scale, what things come from scale and what things don't. And then I can talk a little bit about synthetic data because I think that's a really important question. And it's you mentioned, you mentioned AlphaGo, which by the way, is definitely not the only case of synthetic data or reinforcement learning in AI. But I think it's actually a great example in contrast to talk about where reinforcement methods and synthetic data generation work and where they're hard. So to talk a little bit about scale, it feels like these systems went from like knowing nothing about the world and not really being able to do anything contextual or sophisticated with language to just seeming to know everything that people know in such a short amount of time. It feels explosive. And, you know, obviously in terms of the artifacts, it is like these systems are vastly more broad and flexible and contextual than they were a few years ago. But one thing that I think is important to notice is that what's driving them is really the web, right? They are distillations of all of this information that humans have written down about everything that humans find worth talking about, which is basically everything we know. And so the web did not spring up in the last few years. It's been slowly accreting for decades. What we found is a way to compress that in a queryable way and a remixable way. But the initial scaling, the sort of explosive growth of systems that just seemed every iteration so much smarter, a lot of that just came from being able to tap into that data, which partly meant scaling up to use it all because there's a lot of it. It meant making the models big enough that their parameters could hold the information that was being fed into them. It meant getting enough compute that you could do the translation between the declarative data you're training on and the appropriate learned representations and the weights. And all of that together unlocked the potential of that data. But, you know, people talk about data walls. There is data. Like we only have so much Wikipedia and eventually you can only extract so much juice from that orange. And that's why you're seeing that systems as they scale up, get diminishing returns. And when people look at technologies, at the beginning, it always looks like this. Everything always looks like an exponential curve. And people have a tendency to look at that and assume that will continue forever. But those exponential curves are almost always S curves. Right. It always looks like an exponential. It always turns out to be an S curve. And the next step of progress is switching on to some new idea. And we can talk a little bit more about how that's happened historically in AI. But I think that's where we're at now is you get to the point where like, ah, train on more web worked until we can't train on more web because there's no more web. What are we going to do now? We're going to do some reinforcement learning. Are we going to do some reasoning models? And there are a bunch of ideas out there. Is the world model of the currently kind of like seem to be the best available path forward? Or do you think there's other paths that should be taken as much seriously as that one? I think there's a lot of ideas and they're good for different things. To the extent that you want to continue on unlimited scale at unlimited cost and like intelligence through capital incineration. Sure, the next big thing after the text web is the video web. And let's try to crunch that down. And then all the telemetry from all the internet of things and like more and more data. And that is a thing you can do to continue scaling up. One of the things that we've seen though is for, for example, making systems reliable for making systems follow policies for making them, you have to like guarantee certain properties. That is scaling up in this way is an incredibly inefficient way of achieving that and hasn't been particularly successful. So I think if what you want is a better awareness of, you know, the spatial world around us, the ability to reason about complex physical mechanism short, definitely. But if what you want is reliability and determinism, I think you want a different set of techniques. So one of the things that I think we're also seeing out there is models that are constructed in different ways that are able to have different performance characteristics. That's the space where we fit in. And with our models, you don't need to learn in this very expensive way. So for example, let's imagine that you wanted to learn French and you were going to learn it just by reading books and you were picking up books in written in English and you would notice, oh, here's some French, this character speaks a little French, this guy and you, you're like devouring thousands upon thousands and picking up these little bits of French as you go. Well, eventually this would work. You would eventually see a whole lot of French, but you know, you pick up one or two French books and you're going to be further along. And so the sort of data efficiency or sample complexity that is associated with a given kind of data, a given kind of model, a given learning mechanism can give you vastly different performance curves just because you can get there in the limit of infinite everything. Doesn't mean there's not a much, much better way. So for practicality of people sitting out there building stuff and where most people, I would imagine use Claude, Gemini, whatever, you think that there already now should be a, I guess, more of a strategy of saying, let's just look at what the world of models look like and then figure out what is the exact use case we have for our AI use and then decide if there's a better model. And I'm not sure that that is even happening right now. Is that a fair assumption? I think for some things that is like, that is absolutely the right strategy is to say, what are what performance characteristics do we need? What do we not need? What kind of model exhibits those performance characteristics with the optimal efficiency of data efficiency or compute efficiency at deployment or availability of compute nodes or whatever it is that is your constraint that you care about. And there are going to be some problems that are best solved by just bigger and bigger models trained on bigger and bigger things. Those are problems that involve breadth, problems that involve sort of very complicated, contextual understanding. But where you have problems of determinism, reliability, truth, like that is not the best attack we have today, which doesn't mean people shouldn't be doing it. That's why people are doing that in terms of pursuits of unreliable general intelligence and reliable specialized intelligence are going to require different methods. I was still curious about the small team making and it seemed to be such for an entrepreneurial point of view, it seemed to be such a big opportunity that you can find a specific area where you can create a specific model. You're not that tiny, right? Because you're basically saying I'd like to make a model for people. Smallest relative. Yeah. So like I understand that. But it's just curious of the opportunity from an entrepreneur to kind of be now looking at you can actually build a foundational model. You can't compete with the open design of the world because that race is probably done. But you probably could go out and find a bunch of use cases where a specific model will be very useful and then go and make land. Yeah. I think because of the successes on the axes that benefit from scale, people are very much now thinking about scale, scale, scale. And, you know, obviously that requires a ton of capital, a ton of compute. It requires big teams because anything that's scaled up requires a whole bunch of support structure. But again, our models work in different ways. And the focus of our model is not extracting sort of the full breath of human knowledge from the information as found. It's, as you mentioned before, we're working on a specific class of kind of interactions, which I would characterize as interactions where you have a person who has all the kind of contextual situation that is relevant to human language in terms of having a conversation, referring back to things that have happened before. On the other side, you have a set of functionalities. You can call them tools or APIs that have logic behind them and ways they can be changed together and semantics that govern what flow of information through those tools means. And it's the person talking to this orchestration of back-end functionalities. And that is a huge class of interactions that share a bunch of properties. They need to be reliable. Like if you say you want three tickets, it says here's three tickets, but it's secretly only booked one. Like that's bad. You're going to find out there's going to be a high cost to that. So on top of that, from the standpoint of the APIs, like there are going to be policies and rules and ways in which these things can and should be used. And those rules may change and they need to be sort of specified in ways that are changeable and declarative. And so this is just for this kind of interaction. We just noticed that the existing models are not only expensive. They're not very good at this and that being able to build a model that is better, there's an upside to it also being smaller. But you also just can build a model that's better when you architect that model fundamentally to be structured around these sorts of operations. And that's the approach we took. I think it's really important. Like right now, there's really two kinds of companies that are dominating the market just in terms of like number of companies operating in these ways. One is the companies that are very, very big. They're doing everything at like the most massive scale imaginable. And there's companies building these thin wrappers where like, what is the technology there? It's like, it's probably some prompt. And I think, you know, I guess you have a very appropriate title to this podcast. I think we are trying really as a community to figure out what is beyond the prompt here. And for us, that is models that have additional control surfaces that have additional characteristics, performance characteristics, reliability, the ability to guarantee certain kinds of behaviors. I mean, in building things as a society, one of the most powerful tools, you teach CS 101, right? The most powerful tool we've had historically for building large reliable systems has been modularity, the ability to take pieces, work on them independently and say this piece, we're going to work on it, but we're going to guarantee you that this kind of input produces this kind of output. And there's a contract and there's an abstraction. And this has been one of the biggest challenges in this AI age is LLM has come with absolutely no contracts beyond you will get tokens out if you put tokens in. And this is one of the key things that it was clear to me needed to be different for a specialist model that was going to be deterministic. There needed to be guarantees that you could make about what goes into the control surface. You need to be able to say how that relates to what comes out on the other side. If you want to be able to build reliable things out of it. So we started there and we started thinking about what kinds of systems you could build that let us do how are they structured, what kind of data do you need? And now we're into the synthetic data training. And it turns out that the amount of data you need to get a certain behavior, that sort of sample complexity question can be different by orders of magnitude. It's like learning French from a French book versus incidental French uses in English novels. It's just a very different scale characteristic and it's a better operating curve to be on. It doesn't mean the other approach doesn't bring you gains in different cases. Can you tell us maybe as just to make it very easy to imagine what's a quintessential deployment? If you look at this is a textbook deployment of skilled cognition. Who's the user? What are they trying to do? What's the impact to their workflows or life or business? It's a great question. So I would say the textbook deployment is again the class of conversations. If I put abstractly is conversations between a person on one side and a bunch of APIs on the other. This might show up, for example, in an enterprise to a customer where the customer is doing, maybe it's a customer support kind of thing or changing a flight, changing a hotel, making a purchase, getting a refund, whatever things like that where the person comes with all of the context and everything they want to express is in language. And on the other hand, there's a whole bunch of APIs that can handle this. Like who are you? How are you going to get authenticated? What's your purchase history? Okay, what exactly are we talking about in your account balance or whatever it is? And navigate all of that stuff in accordance with policies. For us, our typical partner is going to be an enterprise. They want to build an agent. This is fundamentally an agent model. They want to build an agent. And it's very important to them that the system be reliable, policy compliant and also secure in a variety of ways. So we didn't talk about this in our system, but the way we've built it, let's us make a lot of kinds of information, compartmentalization guarantees about where that information will go and what can and can't be leaked through training data, things like that. And our system has a bunch of advantages there. So our best customer is some enterprise that cares a lot about not making mistakes, not having things like hallucinations or policy violations. And where essentially the conversations they have, they want to automate them, but it's high stakes to get it right. Now, so far, pretty much all the enterprises we talked to feel like it's important to get things right and that their conversations, their customers are high stakes. But if you think about finance or health care or cases where not only do you not want to mess up for your customers, but where the consequences might be health consequences or financial consequences or regulatory consequences, then there's even this even greater sensitivity to wanting to make sure that the systems are doing what you've instructed them to do, that you have audit traces for that, that you have control surfaces, that you can change the behaviors if you need to change the behaviors and so on. Why do you think that a OBEI, when they launch their health care GPT recently, what would be their way of thinking about this? Because clearly they think that their system will work just fine doing it. So first of all, you also have this issue that any company with a big hammer is going to go treating everything like a nail. And so you can absolutely take a generalized probabilistic intelligence and try to do something specialized with it in the same way that I can train a person to compute square roots. But my calculator will do it faster and more accurately and with a whole lot less energy use. And so depending on the problem you're trying to solve, there are going to be multiple approaches to it. And constellations of non-deterministic models, clearly people are trying that. Now you actually do see a lot of news articles about these sorts of things face planting, either because they're not reliable or they instead of following your refund policy, they follow some refund policy from Reddit in 2005, or you get them off topic and suddenly your customer service agent is talking about something you absolutely do not want screen shot and shared. And so there are failure modes to these systems, but you know, it's certainly you can chain these things together and make a go at it. I just don't think that's, it's not going to be the most reliable way. It's certainly not the obvious way to me. It's just if that's the only tool you've got, that's what you do. And for a company that either is building these big models or is wrapping them, that's the approach they're going to take. And for what it's worth, if you're a big model company that sort of sells by the token, you're probably okay with mechanisms that require spending tokens to check the other tokens, to check the other tokens. Like it's just token use. Same thing with, by the way, with, you know, when people talk about reasoning models, reasoning is a big and maybe ill-defined class of models. But a canonical kind of reasoning model would be just to give a caricature is like run the thing 10 times and then look at what you've got and pick the best one. That would be a kind of like reasoning for maybe a problem that's like a lock and key verification kind of problem. If you're the person who is paying for 10 times the compute, you don't love that as a solution. If you're the person who's being paid per compute, this sounds amazing. And so I think some of these factors are played to. That's hilarious. The unreliability is actually a feature for the model provider. If it's being paid on number of at bats. Totally. Absolutely. When you teach students about AI, what is the most important thing you think that they should know as they leave kind of like the cause? If I had a compact answer that my course would be a lot shorter. I think that answer has changed. So when I started teaching AI, something like 20 years ago, one of the things that we did early on is we showed this checklist of things that humans can do, putting like playing chess or going to the supermarket. And we'd sort of like ask the class like, okay, raise your hand. You think a computer can do this. Do you think a computer can do this? And can a computer play go? Can a computer drive a car? And in the past 20 years, it went from mostly no to mostly yes. And so at the beginning, we would really focus on these core ideas of like, what is AI? What are the kinds of problems, you know, deterministic versus non deterministic adversarial versus cooperative, single agent versus multi agent. And when we talk about these different specialized kinds of problems, which required specialized solutions and specialized representations. Back then, the reason why you would have some people worked on computer vision and other people worked on natural language processing, other people worked on robotics was because to get any of that torque required incredibly specialized representations, incredibly specialized algorithms, different kinds of data, different kinds of learning, you know, everything was different. And we would focus on understanding that kind of breadth and understanding what unified it, which at the time was this, you know, this notion that, you know, artificial intelligence, what is an agent and agent is a system that makes optimal decisions, given its information towards its objective function. And we talk about that. And we still do all of that because that's all still relevant. But now, a couple things have happened that are interesting. One, what you could talk about is just a lot more uniformity to AI. As a natural language person, I think it's great that we've decided that language is kind of a good operating system for AI, but very much now, this sort of, if not an LLM, then at least the underlying kind of transformer technologies are being used kind of very broadly. And so, of course, one of the things we talk about now is that sort of thing. But one of the things that I think is very important now that we didn't talk about before is to start getting into these large scale societal trends. We started talking about digital literacy. AI is going to have a huge impact on how people learn, how people work, what jobs are available for people. And when the key problem in AI was that nothing worked, except maybe like some game playing here or there, we spent a lot less time on that than now, when the technologies downsides are like, we could have a whole class and do on those sorts of things. On literacy itself. Yeah, I think I saw a stat actually from a online education platform, so perhaps slightly polluted as a source of truth. But something like 1% of enterprises are investing in skills. It's just, you could say digital literacy or AI literacy. But why do you think that is? Why the, I mean, 90% ish are investing in the tech, but a small fraction seem to be seeing that there's a literacy problem here. Why is that? Is it again, going back to the kind of objectively it seems like it works kind of a thing? I think there's a couple things going on. I think there's more like, Thomo attached to missing the wave on the technology, like enterprises are being transformed. You don't want to be the only one that's not. And that feels like it's about the technology. I think people have underestimated the skills and human training aspect of this, that whatever technology there is, you can get more or less out of it, depending on how humans engage with it. I think also people just like society as a whole has underestimated how poorly the instincts we have to digital literacy translate, right? Knowing how to find information with search is actually pretty different than knowing how to validate information that comes out of a chat system like chat GBT. One way to think about that is people who were doing a lot of writing or researching now are doing something that looks more like review or editing. And the difference between being a writer and being an editor is really big. But if you're used to being a writer and now some large language model is writing your email for you, you don't think, ah, now I'm an editor. I don't have those skills. You think, oh, it just did my job for me. I can just press end. And I think it's going to take people time because this technology has appeared so quickly. I think it's going to take people time to realize that for all the skills that maybe are less necessary, there's an equal number of skills that not only are they very necessary, but we're not good at teaching them. We don't even have good names for what they are, right? What is the skill of taking a fluent looking output and distrusting it? Well, one skill that's a little bit higher order, but I think is a similar kind of a challenge is the skill of delegation. Or you could even say management. I think it's funny for all that we talk about AIs as assistants, how few people have actually ever had an assistant, right? So how do I work with an assistant? Well, I'm learning with this chatbot. I'm getting my on the job managerial training with a chatbot that's sycophantic. And right, it's not that's not a recipe for success, you know, absolutely. People aren't going to AI literacy training in the same way that there's new manager training, right? How do you delegate? How do you verify? How do you check people's work? How do you mentor? Right? Those are all those are things that typically professionals learn over the course of a career. And now we're all given intelligence on tap and the problem is actually people don't know what to do with an assistant, let alone a capable, you know, junior employee, right? Yeah. And the optimistic take on this would be, well, people will like learn these things over the course of a career. It just hasn't been a course of a career length of time that we've been working with these systems. And there is more going on, because I think a good, if you're working as a human with another human, and you're trying to delegate to them, you do trust them to come back and say, well, I actually couldn't find this information for you, or I got blocked as opposed to, I couldn't find it, but here's my best wild guess, and I'm not going to tell you it's a wild guess. That would not be good behavior from a human, but we see it all the time for machines. So I think there is the problem that you mentioned, which is that people may not have the skills to delegate and manage humans. And then there's the additional layer that these systems do not act the way a human does in a delegation context. Especially as far as the sort of metacognition. Say more about the system doesn't act as a human does in the delegation context, because it was a little garbled at least for me, I just want to make sure, because I think that's a really important point. To me, I think a lot of this boils down to something called metacognition, which is, I think if you had to put a finger on what systems don't have today, you know, we've been talking a lot about reliability, which is ultimately the feature that they lack today, determinism, reliability, whatever you want to call it. If you think about them as cognitive systems, the thing they're lacking is metacognition. In humans, we don't just think we think about those thought processes. When you ask me a question, I stop and I think, do I know the answer? And maybe I do, or maybe I don't. If I don't know the answer, I may make a decision to bluff. I may make a decision to just keep quiet or to change the topic or like, I get to decide what to do about that lack of information. But the fact that I have an explicit representation of whether or not I have the knowledge, knowledge about knowledge, cognition about cognition, this is metacognition, systems don't have that. Back to the example of, you know, of what's the population of Berkeley, right? It's just cranking out tokens, if that's coming from the parameters, whereas a database would be different. A database, you would do the query and you get the answer and you display it, or you don't get the answer and you say, entry not found. So the database does not have the breadth and the contextuality and all of those other kinds of specifications the AI system has, but they are in some sense more metacognitive. They know whether or not the information is present. And ultimately, you know, full intelligence requires both. And LLM today lacks the metacognition. So this is, and to your point about the person in the delegation relationship, they would, what you're saying is they have the metacognition or the self-awareness to say information not found, right? Effectively, right? So manager comes to me, hey, can you do this? You go, I studied biology, not physics or whatever, right? Like, I don't know how to do that, right? Now, I just want to push back a little bit or at least explore kind of the resistance. I had an experience just yesterday with Claude as an example, which can kind of serve as that information not found. Because I read Ethan Molex' post about giving Claude code an assignment to generate a new, you know, $1,000 a month business or something. I just, I thought, that's kind of fun. I just grabbed his prompt, dropped it in Claude code. And it was kind of interesting because Claude, you know, immediately came back to me, Jeremy, I got a level with you. There's no such thing as a business that generates $1,000 a month with no effort. Now, what I can do is this, and to me, it's, if it were purely a function of sycophancy and one next token prediction, I think it would just probably optimistically say, oh, you could do this. The fact that it kind of, to me, that was an example of pushing back. How do you square that example with the definition we've kind of been working with around what language models do? So, I would say there are three levels where what you're talking about happens. And what I've been talking about is sort of the caricature, the simplest version of a completion engine. Systems are evolving, right? And systems aren't just purely trained to produce mimicries of WebText. There are additional steps of training. There are things like alignment training, instruction training. If you do something like RLHF, and you are showing the system, okay, in this situation, I don't like that you did this, I like this one better. And that training does happen. What does that training teach the system to do? Well, it teaches whatever you told it it was supposed to be doing. And that certainly governs style. So, if you talk to one of these models and you get the stylistic, like, that's an excellent question, and it gets to the heart of the matter, like, you're like, how many times have I read that from a model? Well, that's coming from how they were, like, told in their post training to answer, right? And so, if they're not just memorizing WebText, they're also memorizing the post training. Most relative interaction, yeah. Yeah. And so, you can be taught to give those excellent question kind of things. That's certainly not coming off of Reddit or whatever. And so, there is more training. And as a result, when you say, okay, the system said, there is no business that will give you $1,000 a month. Well, where did that come from? It could be there's a web page out there that's like, why businesses won't give you $1,000 a month? Like, it could actually just be that it is regurgitating something that's out there. Like, if you ask it how to do time travel, and it's like, you can't do time travel, there's a ton of web pages that say that. Or it could be coming from some explicit post training, which is like, when people ask the stuff, tell them they can't do it. When they ask the stuff, tell them they can't do it. And, and this has been told to answer that way. So, it is still regurgitating its data. And as these systems get stronger, they will increasingly have a system checking, like, you know, these systems will increasingly become, if not fully met a cognitive, they will start to have those sorts of behaviors. It's very difficult from the outside to tell if the system was aware that it didn't know the answer. Yeah. Or whether it was aware that it's supposed to say it doesn't know in that specific context. And by the way, like, it's not that you can't build a system that looks up some answers, like, rag systems look up answers and then decide what to say. So, they do have two layers. It's kind of what was designed, right? We were talking to the general, one of the gentlemen who wrote like the attention is all you need papers. And I think his point was kind of exactly that, that the whole thing was made to just answer in a way that you expected it to answer. And that was almost kind of like the objective itself. I know we're running a little bit short time. I have one short question if you have time for it. Absolutely. We often asking people like how they use it personally. When you use models yourself, are you a clock code, GPT person, or do you yourself now use different models in your personal work? Yeah, I mean, I'm going to give you a boring answer to this because the truth is kind of boring, which is I do use a lot of things because I want to know what these systems do. I want to know where their strengths are when their weaknesses are. I would say the biggest, if there's something about my experiences with large models that maybe differs from a lot of people I talk about, it's I'm often asking questions to which I already know the answer and like using that as a way to check like, all right, well, what came back? How much of this was right? How much of this was wrong? And then when I ask a question to which I do not know the answer, I sort of assume that the accuracy rate is comparable. You're kind of you're calibrating. Yeah. I'm always trying to calibrate and I am continually impressed by two things. One, the breadth and plasticity and flexibility of these models. It's still just like, amazing. I mean, like I said, when I started this, it was like, will we ever be able to find the verb reliably in a sentence? And now I've got the system that I can ask it about quantum mechanics and get an answer. Now, I'm not an expert about quantum mechanics, but when I ask it about things where I am an expert, the answers come back like, sort of right, but almost always there's like a critical flaw in the answer. And I know I can't find the equivalent flaws in other areas. And that has really impressed on me how important it is to recognize like all of us, no matter how much we know this information, all of us are susceptible to looking at systems outputs. They're fluent, they're confident. The parts we do understand look correct. We assume that everything else is correct. And that's not always true. I think that's a good lesson for everybody to kind of take with them and maybe try to test their model or choice on something they already know. One thing I do like the already know. The other thing is like the task that you would, that where there's not a right answer necessary, or I don't know how to classify this. So I'll give you the exact thing and then you can extrapolate. But one thing that I thought was pretty cool. I had an interview with Time Magazine recently where I was being interviewed, not being the interviewer. And I just on a whim, I took the transcript and I just said, hey, Claude, I've got a chief of staff that's kind of trained on a lot of my blockposts, things like that. I just said, hey, how'd I do? And it was so deeply insightful. And then I said, how should I end? And part of its critique was you treated this interview like a keynote, not like, when you're being interviewed, your job is to be the sous chef, to help the master chef put the ingredients out. And you made this journalist's job more difficult, not more easy. You rambled, you buried the lead, you followed what you're curious about, not what they're curious about, like all this stuff. And I said, great. Now can you help me prepare for the next, because I had another media interview and Claude said, sure, give me the email that the person sent you. I gave it to you. Boom. And I mean, we're talking topics to avoid. Things that you will want to talk about that this interviewer is not interested in. I actually kept it on the screen during my next interview. And so to me, that's like, it's this whole other class of capability, right? Which is I don't have the means to have like a comms expert on my team, right? But now I have a comms expert or at least a reasonable approximation. And by the way, even if it's not great, it does give me more confidence than the alternative, which is zero, right? Which is kind of fascinating. I don't know. I'm just kind of riffing out loud, but I don't know if that's true. No, this is totally right. And I think this gets at what you asked earlier about where are these systems strong. And this is exactly where they are incredibly mind blowing we strong is the ability to take all that information, cross reference it with like all this context that's out there in terms of how you should run media that's a scatter across the web and to be able to distill things down. That sort of contextuality and breadth is just stunning. And in this case, the fact that it could read it all, pay perfect attention to it all, then give you something that if it were wrong, you would know, right? And that you were asking it for its generative capabilities, you were asking it to take all of the stuff, mix it together and give you a synthesis. Like that is a place where these technologies are incredible. It's just that, you know, that's not every use case. But when you have that use case, that's a that's that really lines up well. You were in the loop, you were being the editor, you're like, Oh, I love this, I love this. And maybe there's something in there you weren't going to say, but that's okay. Because this gave you a lot I could draw an analogy to early machine translation, where like, you could read some of it, and then there was some stuff that wasn't really even in the language you were expecting. And like, you didn't get everything, but it was as you said, better than nothing. And in situations where something is better than nothing, mistakes are going to get caught by the consumer. And where the primary product you wanted was this novel synthesis. Well, that's great. I mean, we got a hallucination, it was just a really useful one for you. Yeah, that's okay, gentlemen, you're out of tokens. You're out of tokens. Jeremy Hadley. I put this one more in the, in the hot takes category, probably actually, and Rick, what do you think? You know, one thing, the biggest takeaway I think I had from it is, I remember once I was reading a finance newspaper that we have here in the scanning market, and they were writing about some startup stuff. And as I was reading, I was like, Oh, they really don't know anything about startups. I really don't like everything. The way that everything's phrased was just very, it was very clear that they completely misunderstood everything about startup and financing and stuff like that. And then I was like, Hey, wait a minute, if there's this wrong about this thing that I know a lot about, I wonder how wrong they are about other stuff too. And I think what he was just making was the exact same point with models. And I hadn't really completely thought about it because I love using these models. I use them increasingly, kind of like, I trust them one more just to give me the right answer. But he made this point of saying the population of Berkeley is, and you know, the model will basically have a guess, right? Like, and then I'll put something out, it might be right, but it's probably wrong. And then so he was adjusting that you should write something, you should ask him something, you know, a lot about, and then kind of understand its limitations. Just to calibrate. Yeah. I mean, to me, I think the one caveat, can I put one caveat there? Because I really agree with that tactic and actually love it as a starting point. Say, have a conversation about something that you know deeply to explore the kind of jagged frontier, so to speak. The caveat is I would never recommend someone have a single shot interaction like quiz AI, and then see, oh, is it's response good? That is kind of, that is demonstrably poor AI collaboration behavior. I think what you want to do instead is like, is engage and respond and give mentorship and feedback and guidance and recognize, wow, the AI is fully capable of evolving its understanding based on what I input. But I think that was kind of, that's leading to a second point, which something you always talk a lot about that you shouldn't kind of be working on AI, with AI, you have like this phrase about just making sure that this is a umdirectional kind of thing that you go back and forth with. I think what he was talking to is that people have to change how they work, how they think about what their kind of function is in a workflow. And it used to be that, you know, in going back to journalism, you can be a writer, you can be an editor and it's completely two different jobs. Now, a writer, most likely don't know how it is to be an editor, and an editor might have forgotten how it is to be a writer. And increasingly, though, when AI is basically giving this world of abundance, you now have to teach yourself how to be an editor. And I think you were kind of talking to an editor's function, which is, it just told you how many people live in Berkeley, what you would normally go like, Hey, I wonder if the writer has actually checked that, right? And so you go back and say, Hey, could you please double check that number is legit? I would point the model, obviously, would do a web search and then probably come back with the right number of corrects. But that is, I think, something that most people are not trained to do. And so I do think that the conversation had this interesting kind of like insight from me is that we are just kind of used to work in a world of scarcity, and now we're going to be living in a world of at least genitive abundance. And we then have to train ourselves to be good at being stopping about really want to have excellent outcomes because it's going to be easy just to take the first one. We have to be stopping to be an editor of things that comes at us because we have to be the one who would basically do the fact checker, because the model will obviously just say whatever and all those different things. And so I thought that was pretty interesting to I think it's worthwhile paradigm is his discussion around digital literacy and seeing AI fluency as an extension of digital literacy. So that's a powerful thing. And then furthermore to realize that some of the ways in which we have learned digital literacy are actually a disservice to us. And comparing Googling versus working with AI, things like that. I thought it was fascinating to think about the fact that so few organizations are investing in skills. And I wrote something down here. Let me see if I can find it. Yeah, this idea of new management training is something that organizations provide folks who've been given teams. Nobody's getting new manager training now that they've been given the AI team. I think that that's a really interesting paradigm. And knowing, for example, how to delegate is itself a skill. And if you are a poor collaborator to AI, or if you don't know how to delegate, then you're not going to be able to be successful in a collaboration with this intelligence, not because the intelligence isn't capable, but because you like the managerial skills to derive the best possible work from this new teammate. I thought that was pretty interesting. A small thing that he brought up, which also I think is just interesting, is that we are trained to spot things that are untrue on the normal web, partly because I think we're visually trained. If you go into a website where there's endless amount of banner ads and pop-ups, I think your brain now just goes, this is probably not a very legit website. But we are not, we're used to the things that comes over in a chat window is kind of probably from a friend, or it's condensed. And so things that are written out that is lengthy is something that is real thought through, because that is what we've seen before. And so I think there's also like short hand, in the visual kind of the interface design, there's something seductive in the chat that makes us think that is real. It triggers our impression of thoughtfulness. Or he said, I think the word he used was fluency, but the appearance of truth is something that's difficult for us to discern when we are in edit mode, or especially if we don't know that we're in edit mode. I think it's pretty cool conversation. I think it's a unique conversation in our canon in terms of really focusing on understanding hallucination. What is it? When is it a feature? As he said, sometimes hallucination is the product. It's actually the point. And when you're not looking for a regurgitation of something that already exists, which you're actually looking for something new, de novo to be generated, hallucination is a feature, not above. But then there are other times when your goal is retrieval effectively, that hallucination is a problem. And thinking about when is what system best? I love his point that if a reasoning system effectively just solves a problem 10 times and selects the best one, the person paying for the 10 at bats probably doesn't like that. The person being paid for 10 X the token usage probably loves it. I think it's slightly a cynical view, because as far as I understand it, model companies are largely losing money on reasoning queries and things like that. So I don't think they're getting rich off that. But I think it's an interesting paradigm to realize. His example of learning French from a French book is very much better and more effective than learning French from incidental French phrases in an English book. That was good too. Awesome. I think that's conclude the conversation today. So Jeremy, you do the backing at the end of this episode. What we need, folks, what we need are enthusiastic shares and reviews. I was reminded of Henrik, I'll just make a plug for a former episode, our conversation with Muhammad Ali, the head of consulting at IBM. I was thinking about some of the things they've done in terms of treating themselves as client zero, proving the economic impact, deploying teams to internal workflow optimization, redeploying freed up labor to revenue producing lines of business. It's a really cool example. And if folks haven't heard that interview, they should go back and listen to that as well. But with that, as always, only one thing to say, and that is bye-bye. Bye-bye.