Ship It Conversations: Mat Ryer of Grafana Labs on AI Observability, Agents, Evals, and Operating AI in Production

AI demos are easy. That is kind of the problem. You can wire an LLM into a tool, give it a few prompts, let it call an API, and suddenly it looks like magic. It writes queries, it explains dashboards, it investigates issues. It might even suggest a fix. And in a demo, that feels great. But production does not care about demos. Production cares if the answer was right. If the agent used the right tool, if it leaked data, if it got confused by logs, if it made things worse after a deploy, if the cost graph quietly turned into a crime scene. And that is where this gets interesting, because the hard part is not proving AI can do something impressive in a controlled demo. The hard part is figuring out what happens after people start depending on it. When it is connected to real tools, real telemetry, real production systems, real costs, real failure modes. At that point, AI is not just a feature anymore. It is another thing you have to operate. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick DevOps, SRE, platform, cloud, and security news recap. In between those, I do conversation episodes with people building and operating systems we all end up depending on. Today, I'm joined by Matt Reier from Grafana Labs. Matt is Senior Director of AI at Grafana, a longtime Go developer, author, open source contributor, and podcast host. And in this episode, we talk about what happens when AI moves out of the demo booth and into production. We get into Grafana Assistant, AI observability, evals, LLM as judge patterns, telemetry costs, open telemetry, agent guardrails, and why a chat interface is not enough if the system underneath is making decisions people need to trust. One of the threads I liked most in this conversation is Matt's point that AI gives us a new primitive. That sounds big and abstract, but his example is pretty simple. A conversation plus a loop suddenly becomes an agent. Or, as Matt put it, four loops are back. But once people rely on that agent, the problem changes. Now you need to know whether it helped, whether it answered the question, whether it picked the right tools, Whether a prompt change improved one workflow while breaking another. Whether your AI feature is actually observable enough to operate like the rest of your production stack. We also talk about why AI observability is not just latency, logs, and HTTP 200s. Those still matter, obviously. But now you also care about behavior, cost, tool selection, user feedback, eval results, model versions, prompt changes, and whether the agent is producing something useful or just confidently wandering around your telemetry. Matt also gets into UX, which I think is more important here than people give it credit for. Because if the AI gives you a wall of text, you still have to trust that text. But if it can show you the graph, deep link you into the right Grafana view, apply the filters, and let you inspect the source data yourself, that is a very different experience. And near the end, we talk about where AI can actually help operations teams today, not some giant six-month AI transformation project. More like, pick something small. Use it to enhance the workflows you already have. Let it write the query. Let it help with the first pass. Let it automate the boring parts, but build the guardrails and feedback loops around it, which honestly fits the name of the show pretty well. Ship something small. Learn from it. Then ship again. All right, let's jump in. Today, I'm joined by Matt Reier from Grafana Labs. Matt is a senior director of AI at Grafana, where he's focused on how AI fits into observability and production systems. He's also a longtime Go developer, author, open source contributor, and a podcast host. And we're talking about what happens when AI moves from demos into production, why observability for AI systems is not the same as basic service monitoring, and what teams should be thinking about as telemetry volume, cost, and operational complexity keep climbing. Matt, thank you for joining me. Thank you, Brian. Pleasure to be here. so starting out i'm curious as ai systems move from experiments into production what are teams underestimating yes well first of all i think it's very exciting this whole space and i think that that's where like i always start with this we are we have a new primitive now we have a new capability something that we couldn't do before and that i think is just a very exciting thing there are concerns with ai and valid concerns um but alongside them are i think is the fact that we are now in a new world and this is a very exciting thing you see people making little agents to solve all kinds of little problems we see people building big agents and big complex ones as we are doing as well at rafana labs to solve more complex and deeper problems and just now like having a a new primitive like a conversation being able to now even just putting that in a loop by the way which which was when things became agentic that's just a for loop that's just like oh for loops are back they're they're so back they're cool again you know and i think uh agents are just amazing just a simple primitive put together with something that we're already very familiar with can produce something new so we're still in that phase and i love i love to see all the little things that people are building um but at some point you do end up with like we have a product in production grafana assistant and people are relying on it they're using it as part of their day-to-day some people have leaned very much so into it and it's really unlocked them from for all the observability stuff they're trying to do so now suddenly we can't just vibe it we can't just feel it out we we've got to grow up and make sure that this is going to work for them and i think people underestimate how hard that is it is quite easy to wire up an llm to anything uh do a few simple tools and it and suddenly you unlock this new capability but how do you really know that's good how do you know when you change it you aren't making it worse in some other place you know we used to have unit tests test suites protecting us from all this we didn't really have that now so how do we make sure when we make changes to it it is getting better and how do we make sure it's even like correct in the first place you know it demos very well like you say we've kind of i think checked off that from the mission like has to demo well for people to care to it demos pretty well uh but beyond the demo yeah how do you how do you make sure it does that and we we have we had to solve this problem in assistant which we built our grafana assistant we launched that uh i think it was last year now um and yeah we've had to solve that problem so at a high level what how do you start with that problem statement how do you uh do you build special testing around that but how do you uh actually solve that yeah so i'd say right in the beginning i think people still vibe testing manual testing i still think is the thing to do in the beginning because it's the really when you're just feeling it out you need to know is this even possible is this feasible does this make sense uh and there's so much innovation and there's so much opportunity to learn that actually you still have to have that same attitude of build things and ship it that's why it's why i came on the ship it weekly podcast because it's all about shipping it so i would i would say don't wait for there to be good evals don't wait for you to have this problem solved before you progress uh but it is something to think about in the back in the background and yeah when when we did assistant we essentially had to build we had lots of different tools of course there's like the normal telemetry that we're very good at agrafana labs metrics for measuring latencies and things like this and we even have metrics for costs measuring the the cost of different operations and things uh logs of course kind of makes sense i think tracing becomes more interesting now because you could think of a conversation really as a as a sort of trace you imagine it in that you know in that model um and and i think so so you get a lot already just of the basics but there are new things you care about suddenly uh you know it's not just getting a 200 back from an llm but now how do we how do we make sure that the content the the way it behaved the tools it chose to use how do we make sure that stuff is right and we did it um partly by vibing tools internally literally we'd have like we've got a handful of them um and various dashboards and things and different databases and different ways of storing things you know just trying to sort of cobble together the solution of this and again I think that was the right approach in the beginning because it's a playground where we can experiment and we can try things we can solve problems without having to think about solving the entire problem of this we can just pick one piece of it and just solve it with a standalone thing so there's a lot of freedom in that and I think the culture that we had in the engineering team when we're building this played a lot into that because yeah we had to make sure that people could try things just solve that particular problem not go for months and have big design sessions and just do this enormous project to try and solve the whole thing After we done all that of course, you then end up in a situation where you do have quite a complex little set of internal tools that are all slightly overlapping and doing things in different ways. They don't all agree with each other. Some of them are agentic themselves. So that's even more getting like Inception where the agents are monitoring the agents to make sure that they're behaving properly um and so then we were able to take all that learning and put that together into our ai observability product which is a new feature that we have in grafana cloud and that's like the uh that's our answer to this and we we we think we've picked the most important things that people should care about and built that into the product um and you can see we're different to some some kind of competitive products we're different in some interesting ways because i think some of them were imagined and we whereas we built ours from the real day-to-day of actually operating the assistant at scale yeah very cool what does production level observability for ai actually look like well you do need evals you need to be able to test whether what's actually being asked for is being answered and we have a uh we have a technique inside where we we can we actually have an llm as a judge looking at a conversation and you can just do this you can try this it works just ask it did you answer the user's request because you can see their first message you can see the path you went on and where you ended up did you answer it did the user have to intervene in any way um were you ambiguous you can kind of ask it these things in and you're dealing with natural language now so it's like you you write these tests also in natural language um and and then we can measure that we can start to measure and start to put some data and some science behind the behavior of agents so evals i think um we're going to be increasingly more and more and more important as we we want these ais we want these technologies to be trusted and to be used some people write them off just because they know that they can be wrong so they can't trust it and so therefore they're not going to use it at all i don't think that's the right attitude uh personally you know the internet the network connections aren't reliable they error all the time and we kind of paper over those errors so you don't really notice it same kind of thing with ai we we have these non-deterministic systems in some ways when it comes to doing investigations and things for like digging around your telemetry that can actually be turned into an asset where the fact that the multiple agents take random routes through things um can can actually be good that's what humans might do if they all take a different approach but then you've got an even bigger observability problem because now this one thing suddenly it's multiple conversations with models not even the same model multiple models often uh certainly different system prompts different sets of tools all this complexity now is um kind of a big mess that you have to deal with and we think yeah simple observability simple uh and new signals as well as part of this is the solution to so that you can just answer those questions and make sure that your agents are on the right track so kind of following up with that a lot of teams instrumented everything then they realized they were drowning in telemetry costs uh what changed yes uh i think that i think we're going to go through that as well for ai so i think yeah you're right like people would just monitor everything it feels really cheap like when you're writing something you just add a counter in code feels like the cheapest thing you can do as soon as you then hit that scale suddenly you're counting you're just counting at scale is hard so um yeah you kind of create this new kind of problems when you hit scale. The same thing is definitely true for AI. I think a lot of people aren't really monitoring much at the moment but when they do and as this increases we'll have the same kind of problem and you know Grafana Labs has this technology called adaptive telemetry which is basically what they do is they look at all the telemetry that you've used, they look at all the queries you're making and then they identify bits where you just you've never you've never queried this metric so we're going to automatically sample it down so we'll just take a sample of it we won't we won't capture all of it and of course you can tune it and tweak it and things and this was really an answer to that problem like the the fact that people's telemetry was just growing growing and growing and um and not just in the history of it but even just the now because people's systems and complexities going up so i think that we'll end up with another kind of adaptive telemetry we have already in our solution sampling so you can choose how often the evaluators will run on conversations um so we're kind of thinking about that already and i think this is the advantage of building this solution at grafana labs because we had to solve a lot of the observability problems already and a lot of the thinking applies directly which is very cool like we have these drill down apps which um basically present the data and you just click and drill into it that turns out to be good for humans but actually quite good for agents as well because it's the same problem there's too much there's just too much stuff there and to to look through it all and find it is hard so how do we yeah the fact that we solve that for humans turns out it's quite good to solve for for agents where does open telemetry uh fit into this ai observability story i think it's going to have a big part of it we actually are talking already with um ted young who was one of the co-founders at open telemetry yeah and we're kind of um i've seen some proposals and i've seen some things where the conversation data gets put into trace data so that they they are stitched together in that way we didn't take that approach because of scale issues that we saw so we haven't got that approach in our solution but it could well end up being some some new standard like that but i just think with open telemetry it's going to just take some time you know we like to move very fast in the ai department in grafana labs and in grafana labs in general in some places you do have to go slowly where where you're dealing with standards or foundational things that you you really need to get them right you're going to go a bit slower so i think it'll take some time but i do think open telemetry should embrace this new world and and see which new signals are common enough that we are going to then capture them we got to make those decisions ourselves when we built our solution so of course we capture you know the the basic things you expect metrics for latency and things we add cost into there using metrics and stuff that that's also turns out to be quite complicated to do but very powerful if you have accurate cost data and and with the way that we have labels and things kind of like the prometheus model you can you know use labels for the different versions of your app or different agent names or something so now you can compare performance not just cost and metrics but the other dimensions as well and if you've got evals running you can compare behavior between different models as well so you can say our version one of our model took a long time to answer this kind of question and it's because we hadn't prompted it much or you know we just gave it one tool that it could use we did some work we added a couple of tools we tweaked some prompts and our our new version is now much faster it solves this problem it's much more accurate users happy we capture ratings as well of users of you know thumbs up thumbs down as well comments um all the sort of practical things you need to actually have some kind of feedback loop where you can improve your agents is that what you have found to be i guess the best feedback loop is like providing like right in the chat the the ability to give feedback immediately like hey this was not the right response or hey this was not detailed enough or it went off on a tangent that was not really related to the question? Or is there other, like, are you feeding that in with other information as well? And then how does that work? Yeah, I think user feedback is the most valuable. And it is nice that you can, even just a thumbs up, thumbs down is enough to give you something to go on. I think the actually typing in a comment is even better, right? Because we can genuinely have LLMs consider that commentary and even suggest prs into our repo to improve prompts and things like this um so mostly though we don't get uh we don't get as much of that as we would like so we can't rely on that and the other thing is we're very sensitive about learning from people's data without them realizing that's what we're doing so yeah we are we are very um yeah we want to make sure that we are uh not yeah just being very trustworthy basically if we're going to build ai that we expect people to use i think the standards have to be very very high for things like that make sure that your data is protected uh it's easier than ever to to leak data uh we we had a case where uh luckily it was test data but we uh we we set up a github permission for an mcp server we didn't realize that you don't need right permission to open an issue in an in a public repo so we didn't give it right permission but still it started to open an issue and although it was a test box so it was no problem it was putting data that it saw in there into the issues very cool feature of course and extremely useful for like irm and incident management and things especially useful when it opens a pr for you very nice but we've got to do that carefully and safely And I think knowing that your systems are safe and aren't behaving and that they will react properly and appropriately to prompts that are kind of probing and trying to do malicious things. I think that all is a very important part of this that we start. We sort of still scratching the surface on. Yeah, I think it also kind of highlights the need for guardrails because LLMs are over eager. right, to solve the problem, get to the root cause and provide a solution. And if it has the means to do that, it will. Yeah, it's true for tools as well. It's true for like, if you have, if you give it lots of tools, it's just going to be like a kid in a candy shop. It's going to be so excited to just go and use all the tools And given that it non you can easily end up in a situation where it using quite random tools to try and solve problems and it trying to use a hammer to open a door for example which i not a diy expert but i don think that's how you do that um but yeah so so how do you how do you sort of tackle that i think that's kind of another interesting uh angle on this yeah so from a guardrails perspective i'm sure at the IAM level, like you're providing read-write permissions or scoped permissions based on what's needed, but are you doing like pre-prompting as well or are you not relying on pre-prompting at all because that's just too close to the LLM and you know it could be influenced? Well yeah, so we have... How do you build that system? Yeah, we've got a few different things. So our AI observability solution lets us write kind of security type tests and assert them basically on the model. So that kind of gets us, we can do some kind of testing there just to test the end point to see how are you going to behave if a user asks for this. You know, it's also dealing with customer data. So it's also dealing with data in logs, which can just be any text. so you have to be careful to make sure that it doesn't just load some logs and think oh i'm just going to execute something in in here um and the real answer in in assistant is we just have lots of different checks and different gates and things happening because there's so much it's a very complex um solution i think it looks simple when you use it and the ux there's another thing i'll get to talk about hopefully is the the user experience of this is paramount here more than ever and it's always been important i think in tech but the um you know the fact that you you the fact that you need to uh just just have well the fact that we have this complex system we use multiple models we've got like specialist tools inside assistant all running grafana cloud there's preemptive work that happens around discovering the infrastructure and so we put that into a semantic vector database so it's the agents have access to kind of learning that it's had before when incidents and investigations are happening that also is a great source of information for agents but again how do you make sure that's all safe so i think at any point when you're integrating and bringing some data in you have that security question of in what ways could this be malicious and what protections can we do here to help and make sure it's not going to be a risky system for people to use. And how have you thought about the UX side of the user experience side of the AI agents? Like how is that architected or how do you think about providing that experience in a quality environment that is actually useful? Yeah, this is somewhere where I think the team have excelled. And we have heard this feedback. One of the big labs told us that this is, at the time, this was the best looking version of something that they'd seen using their models, which was a big, big compliment, of course. I think the key is we on the team just got really annoyed by walls of text. It was a kind of a byproduct of this is that, you know, the fact that you are dealing in conversation means it's generating text as what kind of how the whole thing works. And there's just so much text. Now, when it comes to trusting LLMs, you know, in that text generation, that's where they can hallucinate things. so we instead of using text wherever we can in the assistant we'll use a grafana visualization so you know luckily we have this library of these beautiful tools grafana's been around a while so anything we needed to represent from logs or from traces or metrics or profiles or other other ways with with graph fizz and mermaid diagrams things like this we would ask the assistant to do that and to prefer that and so it's it's really about um i think having people with excellent taste a kind of designer engineers uh in one really um having them obsessed with that mission of we're going to fight this wall of text because if if you know a picture tells a thousand words so if we can show you here's what the data is doing not only do you can you see it as a user that is still better as an experience for users than than reading what someone tells you about that graph right you someone might say oh it's a flat graph is it flat but high is it flat but low yeah you can describe it you know it's actually easier just just show me the graph this is why i think like don't think dashboards are going to go away in the ai world because we're always going to care about what's true and if that if the agents are correct um we're going to want to see that data so the grafana assistant is very visually beautiful for that reason because it piggybacks a lot on grafana um but i think yeah it's it's important to it's going to make a big difference and i think you can you can separate yourself if you put a bit more effort into making sure that that experience is excellent i think you can separate yourselves from similar capabilities it turns out for some reason and we we have a few theories on this but it turns out that um llms are quite good at grafana by default um and we think it's because of so much of its open source grafana the whole code base is open source but also all the community stuff all the discussions around there like community dashboards out there and a lot of the connected ecosystem prometheus and you know open telemetry all these other things are also kind of very open source and and i think the llms have hoovered all of that up so they naturally have quite good context so when we first stitched together the llm uh with grafana and gave it a few tools and had the ui in the in the uh grafana front end it was surprisingly already quite good and this is what people are finding when they wire up claude code with our gcx which is a cli tool that we released that thing uh you know for certain tasks it's very good and if you've got your code right there as well like it's very handy to be working alongside with production code with telemetry you know and having the having them considered at the same time um but all the work we then did in assistant with the prompting the fine tuning of certain little pieces of it the specialist tools and uh lots of other kind of secret sauce that's got in um created this uh complexity that we had to deliver in a simple way so i think if you use assistant it'll look hopefully it just looks and feels very intuitive you know it does call things like it can build deep links so it knows how to jump you to the right page in grafana but not just the right page because all the your because all the filters are in the url parameters it will apply those filters as well so it takes you straight to the right view um and so that was another good thing i think we did was we aren't going to reinvent everything and give you an ai only experience here we are going to use all the capabilities that you're already familiar with and some of the amazing tools that already exist in grafana as part of this too and yeah and like the fact that the assistant knows what page you're looking at little things like that if you go to a page and ask and say tell me about this it knows what you're talking about so that that really matters those kinds of little details that experience that the user's having i think is what makes it feel like excellent, helps you trust it. You see the data itself as well. So you can see if it's made a mistake. And I think UX is more important than ever. Very true. Okay, so for a team that is starting out, wanting to introduce AI, where do you think AI can actually help operations teams today? Like where should they start? What's the most important telemetry or piece of data to start using to analyze? Well, it will depend probably on each of their cases. The cool thing is they will know, they'll have an instinct already for things that they already care about. So the thing is, the nice thing is probably people already are using some kind of telemetry. They're using some kind of observability. And really AI is there to enhance that experience. It's there to make that easier to do. Since it writes the queries for you, You don't have to learn PromQL and LogQL. And, you know, I've used those languages before, of course, but I would always have to go and look up again how to do something or prefer the query builder, honestly. Like now you don't even need to do it at all because genuinely through natural language it can generate those things. And it generates very complex ones as well. It's far out, you know, surpassed me in my abilities. But I would say, like, use it initially to enhance what you're already doing and see ways that you can automate. In some ways, that's really what we're doing. It's just automating it, but automating it in a trustworthy way. And I would say from those few little early things and do stuff, ship it, ship it weekly. I like that actually as a motto, ship it weekly. Not just the fact that this podcast runs every week, but I actually ship things every week. I like that because you're kind of forced to focus on scope. the scope that's most important you know if you if you have a short window for shipping things then you've got to really focus on what's important and i think that applies to people picking up ai things don't do some enormous ai project it's going to take months and months because honestly everything's different by then anyway pick something small do it action it and talk about it and share it uh and see what it see if there's excitement you can drum up it's what happened at grafana labs and then now you see people all over the company building all sorts of things and doing amazing things with agents some of it will eventually be a product i'm sure but it's it comes from play and that's something that we do all right at grafano labs we have a lot of space for people to play and get creative and try things and fail for things to fail um there's a great analogy that i love where they they did a study where they got these two groups and they told they were making pottery you know on the pottery's wheel but they were all amateurs they told one group you've got to make the best pot you can and they told the other group just just make as many pots as you can doesn't matter what they're like just make lots of them and then by the end of it the the group that were making as many pots as they could were making better pots than the group that was trying to make the best pot and i think that tells you a lot i think it's the trial and error that you learn you know there was a lot more waste on the floor of this this other team where where pots hadn worked and they had to throw it away or or smash it up and and tear it down and turn it into something else um so it feels wasteful i think to some but and and they might be tempted to kind of optimize that away but honestly that's often where the learning is so i think people should pick up something tackle it something small have a play see what you can do and from there yeah you create new problems for yourself but they're good problems to have where where do you think ai is going with observability this year i mean obviously it's it's the talk at every conference but is there overarching areas in observability that you think that ai um that we're at just at the forefront like we're able to do now with larger token uh more tokens more uh better models um faster models is there anything that like lends itself now that maybe we couldn't have done six months ago or we could do in six months um based on these these rapid changes to ai i think seeing seeing the models get better and get smarter is nice we are finding that we have to tell it less which is interesting we're also finding that we we are basically tuning our prompts to a specific model we noticed with claude 3.5 to 3.7 even uh 3.7 didn't perform as well as 3.5 by default we had to make it work by changing it and tweaking it because it was a kind of different thing um and so i think we'll get to the point where uh agents are yeah you have to tell them a lot less i think it's their instincts would be good we already have plenty of tools to help with like skills and things you know in assistant you can write skills which lets you basically control how the agents behave this is nice if you've got like expert sres where they've spent years building these patterns and practices and they've got these high standards you can then give those same high standards to the agents and they'll follow your behaviors so you can already customize and tune it like that um but i think they'll that's going to just improve and i sort of expect that to happen bigger context windows um not necessarily not necessarily seeing that improve things too much we already have kind of too much context a big challenge that we're doing at grafana labs is and with assistant when it does an investigation it's about going and finding the right context and keeping the right context and discarding the things that's not right so big context windows kind of they're tempting you just think oh we could just fill it all up with everything i don't think that's going to help it at all uh it's just more of it more for it to think about yeah um but i do think we we are going to see more things automated and i think with with with and one example i think is an easy one is rolling back a deploy let's say you you you kick off a deploy through github whatever your process is at the end once it's out maybe you you know there's some you you do this in staging and things whatever you already have all the great stuff in production have agents actually go and do a series of kind of critical tests check things check all the dashboards you know the they know how to use grafana so they know how to use all your telemetry uh so how can we get all of that and uh make good decisions from it and then a decision might be that that deploy is not i'm not happy with it for whatever reason and i'm i as the agent i'm just going to decide to roll it roll it back so i think that's a safe quite a safe operation because it's you should always be able to roll back a release that should be a safe operation um it's not always so it's you know it's not without risk but um yeah that feels like something like that i think we're going to get we're going to see more and more of that and then i want to go further what else are we happy to let it do can it can it deal with scale issues based on unexpected events can it um what else can it do where else can it take action and i think uh we need to get a bit braver sometimes uh on some of these things it's on us to build the tools and make sure that they are behaving well and you can see what they're doing and you know people will some people will always have the the human in the loop to give the approval and hit the button and i actually think that's also fine but but all the work that leads up to that point could could there's still a big space there to explore and we are exploring it is very exciting some of the things that can happen um but i do think we're going to have agents doing more of this operating things for for us at least the simple stuff and that frees us up to then focus on the cases that aren't simple and that are more complicated or frees us up to to build more things and do more other things yeah very true um okay so wrapping up is there any belief about ai or observability that you think is just wrong i think uh one thing that's wrong is because it's not 100 you therefore can't trust it as we mentioned earlier like um yes it's it does it's not perfect it gets things wrong so do people but you still hire people hopefully for now uh but so i think that i think but that's all that's not just ai that i think people often struggle with that i remember um i remember like in covid times i was wearing a mask and a guy in the elevators like thought i was silly like thought i just believed some hype or whatever so he's like why are you wearing that they're not even effective they only like only prevents 20% of spread or something so it's like oh is it so it's only 20% fewer people going to be oh so but because it wasn't 100% he dismissed it I see people doing the same thing with AI and yeah and like or they think with with coding um like and you know I'm it's not as good as me because I prompted asked it to do something it didn't do it as well as I would have done it and so I did that last year and I'm not going to touch it I, AI is not for me. That's also a mistake. Keep trying them. It's usually you haven't prompted it well enough. If it's not working, you can prompt it differently and there's different things you can try. Uh, and you can also be much more specific. In fact, the more context you give it the better. So yeah, it's not a hundred percent. It's not perfect, but it's good. It's changing the world. Yeah, for sure. Um, Matt, any closing thoughts for our audience? no but just want to say keep shipping it weekly because um you know you've got it that's the only real way to know you're doing anything useful iterate yeah yeah and keep listening to this great podcast as well with the with brian appreciate it thanks matt thank you so much for your time thank you for coming on really really appreciate it pleasure all right that was my conversation with matt ryer from grafana labs the thing that stuck with me the most is that AI in production has to grow up pretty fast. It is fun when it is a prototype. It is fun when someone wires up a model, gives it a few tools, and suddenly it can answer questions that used to require bouncing between dashboards, logs, traces, docs, and half a dozen Slack threads. That part is genuinely cool. But once people start depending on it, the bar changes. Now it needs observability. It needs evals. It needs feedback loops. It needs cost awareness. It needs guardrails. It needs some way to tell whether the agent actually helped or just produced a convincing answer. And that is where I think this conversation gets useful for platform and SRE teams. Because the trap is thinking AI observability just means watching the AI service like any other service. Did it return a $200? How long did it take? How much did it cost? Those are good signals, but they are not enough. With agents, you care about the path it took, the tools it used, the context it kept, the context it threw away, whether it answered the actual user request, whether the user had to correct it, whether the model changed behavior after a prompt update or model upgrade. That is a much messier kind of production system. I also liked Matt's take on trust. A lot of people treat AI like it has to be perfect or it is useless. And I get the instinct, especially when we are talking about production systems. But we already operate plenty of imperfect systems. Networks fail. APIs time out. Humans miss things. Dashboards lie by omission. run books rot. The answer is not blind trust, but it is also not refusing to use the thing because it is not 100%. The answer is instrumentation, feedback, constraints, and judgment. It's the same as most other production problems, honestly. The other big takeaway for me was the UX side. AI should not just bury operators in a wall of generated text. If the system can show the graph, link the right view, apply the right filters, and expose the evidence behind the answer, that is way more useful than a paragraph that sounds right. Because at the end of the day, operators still want to see what is true, not just what the model said. So my takeaway is pretty simple. Use AI to make the work easier. Let it help search through telemetry. Let it summarize patterns. Let it assist with the first pass of an investigation. But do not skip the production discipline. Measure it. Test it. Watch the cost. Build evals. Keep humans in the loop where the blast radius is real. And when you give an agent tools, remember that it may actually use them. That sounds obvious, but it is going to matter a lot. I'll have links to Matt, Grafana Labs, and anything else we mentioned in the show notes. If you enjoyed this conversation, follow or subscribe to Ship It Weekly wherever you listen to podcasts. It helps the show, and it makes sure you get both these conversation episodes and the weekly DevOps, SRE, Platform, cloud, and security news recaps. You can also find everything over at shipitweekly.fm. Thanks for listening, and I'll see you later this week.