Ep 772: AI You Can Trust: When Good Enough Isn’t Actually Good Enough

This is the Everyday AI Show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business, and everyday life. You know what most companies pitching AI agents aren't actually doing? Deploying AI agents. Sage just did the opposite at Sage Future. They expanded their AWS collaboration to put real AI agents into real finance workflows built on Amazon Bedrock Agent Core and available in the AWS marketplace. So that means enterprise-grade AI for your finance team without the IT headache. Check it out at Sage.com. What good is AI if you can't trust it? I think as agentic AI grows at an unprecedented rate, sometimes there's a rush to just do more, create more. But when was the last time you went through and audited your AI? Because if it's good enough, is that good enough? Well, that's what we're going to be talking about today on Everyday AI. For our podcast audience, you might not see this, but we are at Sage Future and I'm excited for today's show. So we're going to jump right into it. So help me welcome my guest live stream audience. We have Jeremiah Edwards, the head of Sage AI. Jeremiah, thanks so much for joining the Everyday AI Show. No, great to be here in San Francisco with you. All right. So we're going to get to everything that's new here at Sage, at Sage Future. But let's start with the big question. When it comes to explainability in AI, when it comes to trust in AI, how can people, whether they're business owners, whether they work in finance, how do you actually get to the point where it's not just a race to do more, produce more, more outputs, but how can you get to that actual trust and explainability piece? No, it's super important. I think focus on correctness and focus on explainability is a big gap in a lot of AI tooling today. We kind of stopped and said, oh, it's amazing that the dog can play piano, but does it sound good? I think I would encourage every business user to look at their AI systems and say, okay, how did it calculate these results? Is it correct? And is that better than the process I was using before? Is it really saving me time? Because there's a lot of noise out there. Yeah, absolutely. And let's actually rewind a little bit and zoom out. So for those that aren't familiar with Sage, obviously a global player in the game, but explain a little bit what Sage is and what you all do. Yeah. So Sage makes accounting and back of office software for small and medium businesses around the world. We focus on that accounting function and give a CFO tools that they need to run a business. Yeah, absolutely. And with Sage Copilot, obviously a lot of new announcements and new capabilities coming out of Sage Future. Can you maybe highlight a little bit what was announced? Because I know there's a lot, right? But highlight maybe a little bit on the Sage Co-Pilot side, the new capabilities that Sage customers now have. Yeah. So Sage Co-Pilot is our user interface for all things generative and agentic AI. And we released that a couple of years ago, but these days there's this big transition from generative, you know, kind of chat agents to agents that can really take action within the software that people are using. So in particular, our finance intelligence agent is helping people close the books faster, keep track of their budgets and answer questions about their business that they couldn't before. Yeah. And so last year, you know, I had a great conversation with Aaron and, you know, at that time, it wasn't that the models themselves, you know, were the biggest hurdle, But, you know, looking at the models from a year ago to the models that power the technology today, you know, there's definitely been a step change, you know, more agentic by nature, you know, smarter, more capable, etc. On the flip side for, you know, people in finance, because, you know, a lot of day to day operations, they start and they end with the numbers. How do you balance that, you know, the rapid growth in agentic capabilities, even within Sage Co-Pilot with the potential for maybe getting that one number wrong? Because if you get that one number wrong and that's your starting point, things can go downhill quickly. Yeah, no, I think there's been this kind of obsession for the last three or four years on the model, the model, the model, the model. And I think that's in one part justified because Anthropic and OpenAI are releasing some really cool stuff every day, every week. It gets a little bit better, a little bit faster. But in order to get finance, bulletproof, accurate numbers, it's not just the model. I think there's an entire system of software that has to come together. And more and more around agentic AI, I think the industry is realizing this and catching up. It's not the model. It's the model plus the agentic harness, plus the tools that it has access to and the data sources that really matter. and the best AI model in the world cannot do your accounting for you if it's not connected to the best accounting software in the world, connected via standard protocols like MCP or A to A. And without that, you have no hope of getting a good answer. Yeah, and I think that's a great point because you even brought up things like the model context protocol and the A to A standard. right and it seems like there's also for people that maybe don't have a background in AI and machine learning and math right like you but maybe business owners in the C-suite that want to be involved and understand how the AI interacts with their numbers can you even explain a little bit and specifically you know even about the math capabilities of these models right because maybe two years ago people you know you see all these kind of horror stories people share something like oh, you know, this model can't add, you know, two plus two, but we're not opposite there today. Can you just give everyone else a quick update on just where these foundational models are now when it comes to being able to understand the basics that you need to have in finance? Yeah. So all the LLMs in the world were trained on a vast corpus of text data, right? They know how to complete text and at a very fundamental level, they are translators. They able to translate from a question to an answer they run into a lot of trouble because there a lot of lies on the internet I don know if you knew Jordan but not everything on the internet is true And errors like that, factual errors are in their training data. The training data sets are so big that that can't be filtered out or corrected very easily. That process takes a long time. So enter the dragon. In order to get correct answers out of these systems, we have to give them access to tools. We have to give them a calculator to do arithmetic. And without that, you're definitely going to get incorrect results. And I think it can be hard to find the line where that training data was good and correct and where things start to go off the rails. So I do want to get back to making sure your AI that deals with the numbers gets on the rails. But I do also want to talk about a pretty big partnership that you guys announced with PwC. And in that partnership, the study that came out that said 71% of finance leaders would reject AI that can't explain itself. So talking about this explainability, there's obviously things at the baseline model level and then at Sage co-pilot level. But can you just talk a little bit about the importance of explainability and trust when it comes to the numbers? I know Like it maybe just seems like it goes without saying, but I want you to say it directly because I think some people need to hear it. No, at the end of the day, the CFO or a small business owner is accountable for for the integrity of their books. You know, when their taxes are filed, when an audit comes up, it's it's not Claude that's sitting in the hot seat with the IRS. It's the CFO. And because of that accountability, the expectations that they have of any software system, including AI, are very high. So if it's not completely correct, it's not good enough. Close is not good enough. And because of that, I think it's critically important that in all of the software that Sage builds, in all of the AI that we bring to market, we think of explainability as a first-class design principle. And what does that mean? It means that every step the AI agent took to answer a question is there. It's fully auditable. It's right there for someone to see, including data sources, including API calls, and including reasoning that these models can bring on top of that. Yeah. And even speaking of reasoning, right? So not to get too technical for our non-technical audience, right? But what does it mean now that the models used by Sage Copilot can reason by default, right? Whereas two years ago, that wasn't really an option for anyone. So what does that mean when it comes to increasing the trust and increasing the explainability when you can see, here's how Sage Copilot got to this answer? Yeah, so I'll use another term of art. I hope it's approachable. But I think more and more chain of thought reasoning is a first-class consideration for anybody using these systems. You know, you're not just calling an LLM once and getting an answer. I hope not. You're asking it to critique its answer. You're asking it to dig in. You're asking it to look at the other tools available and pick the right one. And you can look at, and hopefully everyone wants to surface this reasoning to users, we certainly do inside Sage, you know, what the model was doing and why it was taking the actions that it was. But that's the only way that you're going to get AI that a CFO will use. Yeah. And yeah, FYI, you know, audience, you've heard me be the old man shaking my fist on the chair all the time about looking at the chain of thought. And, you know, just FYI, you know, Jeremiah, one of the smartest in the game, said to do the same thing. But also you bring up a good point, the transparency of being able to look at that inside of Sage Co-Pilot, which brings me to time. It brings me to agency. for people, I consider myself mid-career, right? But for those people, yeah. But for those people that have been in a CPA for a decade or a CFO for 25 years, and now they see these agentic models that can do the type of work that they've kind of built their career around, right? So I think people look at it two ways. one way they say, this is great, right? And I can check the work inside of Sage Copilot. It explains how it thought. Great. The other people are like, wait, what does this mean for my job, for my time, for my agency, for my expertise? I know this is a much bigger question that we could probably talk about for hours, but how do you begin to unravel what this means for your own experience and for that human agency? Yeah. Well, first of all, I want to acknowledge that this is a universal problem. I mean, as a mathematician, as a software engineer, these are problems that I also face. You know, wow, the computer is getting awfully good at what I used to do for a living. Goodness. But no, I think within finance, within accounting, there's actually a shortage of talent these days. Fewer people are coming out of CPA programs ready to jump in and help small and medium businesses get that job done. And I think AI is kind of rising to meet that challenge and fill some of those gaps. And to anyone who's fearful of that transition or looks at this and is like, I'm not sure where this is going, I'd say you don't have to eat the whole enchilada at once. I think you don't have to use every piece of AI out there all at once. And I don't advise that people do. I think you can start with task-based AI. Process automation. You know, I love to focus on kind of AP, accounts payable. for that. People used to have to look at invoices and manually enter data into the accounting system. No human has to do that anymore. We have tools for that now. And that's great. That is AI. A lot of people don't think of that as AI anymore because it's been around for a while. We've gotten comfortable with it. But for anyone who's skeptical of that transition, start there. It can save you a lot of time. I think our customers say it can double the number of invoices they can process every month, which is huge. Getting to your point, though, about how does that change the nature of strategic thinking within finance and the role of the CFO, I'd say it frees up a lot more time for looking ahead instead of looking back. When you're crunched for talent and under a lot of time pressure, often you get into these sort of batch update cycles. We have to look at the last quarter, make sure everything's correct, turn it into an update. And by the time you do that those numbers are out of date And I hoping that AI and certainly the AI we build at Sage can help them you know be more forward looking and really kind of meet this dream of continuous accounting where the data is available. The data is available to the people within the business who need it to make decisions. That takes work off the finance function and allows them to really focus on what matters for them. Yeah. And, you know, you bring up a good point because it does seem like as agentic AI becomes more powerful as, you know, layers of intelligence like Sage Co-Pilot, you know, help it become more explainable and, you know, more utility as well, obviously. you know, you talked about, you know, it will free up a lot of time, but I'm wondering, you know, can you get maybe a little tactical and, you know, so where should people be spending that actual time? Is it, you know, more on the front end? Is it more on the back end? Should we be spending more time? Like, like me, I'm a dork. I love reading the chain of thought and, you know, trying to make it a little bit better or get better outcomes. You know, where should people, you know, specifically whether you're, they're using Sage Co-Pilot or not, but if they are using agentic AI in finance, where should those experts be zeroing their time in to make sure that they're getting the most out of the AI output, but also to make sure that they're still keeping those crucial skills sharp? No, absolutely. I think, let me give a very finance-oriented answer first, and the short version is governance and control. Any AI system that people are working with should give you the confidence you need to, you know, in the data, in the results it's bringing to you. And if it doesn't, don't use it. But, you know, it should also give you a control plane for deciding, you know, using critical judgment to decide what it should do on your behalf. And if the human is not in control of the AI system, you shouldn't use it. Yeah. That's another design principle. But ultimately, I think it's about a point we touched on earlier, which is accountability. If the finance function is still accountable for the financial health of the business, that means tasks are going to be a lot more review-based. AI is the doer. The human is the supervisor. The human is the reviewer. And you have more time to do that because you're not stuck on lower-level tasks. Yeah. And a lot of success probably comes from constantly auditing and improving the people, the process, and the technology. How do you schedule those feedback loops when the technology does change so quickly? How often should you be visiting change management within your financial organization? How often should you be re-auditing some of these agentic workflows that are now possible this year that maybe weren't possible? when I talked with Aaron last year, right? What does that kind of iteration loop look like in the age of AI everywhere being very impressive? Quick pause. Want to know where AI ROI actually lives? It's not the flashy demo. It's the work nobody wants to talk about. At Sage Future, Sage tackled the painful side of finance system rollouts, implementation. Financial data migration, chart of accounts, mapping, configuration, the work that stalls real transformation projects. Sage is using applied AI to make that faster, more accurate, and auditable, with humans staying in control. That's responsible AI doing real work that actually moves time to value. If your finance team is staring down a migration, check out Sage at sage.com. dot com. No, it's I want to answer that question two ways. One is because, you know, as a as a builder of AI systems, I certainly have opinions about, you know, how it can help save people time. But but I'm also, you know, an AI user. Right. We all we all are. We all wear that hat like I'm using AI to try to, you know, make make my workflows more efficient and also, you know, transmit that to my team, also learn from my team. And I think how you do that effectively, you know, is going to look a little different in different organizations, depending on your risk appetite. But I think at the very minimum, I'd advise that everyone be looking at AI tools and thinking about how, you know, how that can change the operation of the team. And I really do love to think about it at the team level. And sharing information is the bare minimum here. I think what naturally arises out of that is some use cases are going to be better than others. And as a leader, you can identify which ones those are, which ones you want to promote within your organization, and incentivize people to be creative, to bring those to you. Because otherwise, I think you end up in a situation where, you know, there's shadow IT, there's people using tools that are kind of outside of your visibility and control, and that's worse for everybody. You know, obviously, you know, vendors like Sage are going to make the best AI integrated with our accounting systems as we can, but we exist in an ecosystem, and, you know, while we can provide the best solution inside inside Sage Intact, for example, you know, like people are going to look at other solutions and use them for different use cases. And so you really have to give yourself the visibility into that and then advise from there. So you brought up a couple great points there, both about shadow IT, right? And which I think is important to, you know, or maybe let's just do this. Can you explain, right? Because I think a lot of people think, oh, I can only get the best models if I go to, you know, Claude or OpenAI or Gemini or whatever. Can you explain about how you just bring those best models inside? I think we maybe skipped over that point, but I think it's important to talk about, yes, you still get world-class AI capabilities with Insage Copilot. No, absolutely. I think we, you know, we have technology partnerships, you know, with all the best players. We do use models from Anthropic and OpenAI and Microsoft and OpenSource. But, you know, by integrating them into our AI platform, by adding a layer of control and governance, we're able to ensure that all the answers that come from Sage Copilot are grounded in real data, you know, and are relevant to what the user is trying to accomplish. And that's, you know, that's really because we have the context within the accounting system and within the user identity of who's using that system to give them the best data and answers we can Yeah absolutely And that does help fight against the shadow IT which I know is always a big problem But you brought up something else interesting that I really like to discuss just the risk appetite Yeah Right So for I think obviously in the enterprise we past that conversation for the most part Right I think most large enterprises that are moving fast you know they got it somewhat figured out when it comes to you know finance and AI But I think a lot of SMBs are still maybe struggling with that risk appetite, right? How much, you know, should we be pushing AI, you know, when maybe they don't have a team of experienced, you know, machine learning engineers that can come in and, you know, help set this up. So with that, for maybe those, you know, small, you know, maybe those small and medium-sized businesses, when they're looking at something like Sage Copilot, and maybe they're still doing it the manual way, right? and they're saying it might be too risky. How do you address that? No, absolutely. I think I approach this whole decision space very much with a mind towards like use the right tool for the right job. You know, you can maybe just use ChatGPT directly for some functions, you know, ideation, creativity, marketing copy, a first draft, not the final draft. That's fine. But when it comes to finance and you really do need to have a higher standard. And I think people are right to look at that risk and be like, wait a second, do I want the robot anywhere near my general ledger? And again, my answer to that is just that it's not an all or nothing type of thing. I always suggest that people look for individual use cases, subfunctions where AI can bring whatever amount of value you're comfortable introducing today. And more than 10 years ago, the first AI that we built at Sage was outlier detection. It just combs over every general ledger entry and says when something is anomalous, when something got misentered, or when fraud can pop up. That should not be controversial. It's not changing anything. It's not posting transactions. It is simply there as a watchdog, as a stopgap. And that's still running today. And now we can do a lot more, obviously, with agentic AI. But even within that system, our outlier detection on the GL is one of the core tools behind our MCP layer, behind our finance intelligence agent that makes it work. That's what grounding means in the real world. Yeah. You know, we're grounding our agentic answers with real anomalies detected with good old machine learning. Yeah. And, you know, you said something important there that I want to point out, you know, doing the best with today's agentic AI. But, you know, obviously you can't be looking at the past if you want to be a future forward, a future facing organization that's keeping up with, you know, bleeding edge AI. So, you know, obviously you can't, you know, lay out the playbook for next year's sage future. Right. But what should business leaders, you know, on the finance side and otherwise be doing today, specifically when it comes to their books, their processes, so they can be ready for whatever comes next? Yeah, I mean, if you live on the bleeding edge, you get cut sometimes. I'm there with technology and our customers are there with their processes and with all of the changes in this space. So my best advice is hold onto it loosely. I think there are some problems that we can solve and AI is not one of them. AI is bigger than that. It's bigger than you, it's bigger than me. And we can orient ourselves around that change and be ready for that change. So I think my best advice is start using AI that you're comfortable with. Keep an eye on all the AI as best you can, you know, through everyday AI and other resources, right? So that you're aware of what's going on in the ecosystem. And then when you do see something that's a good fit, you know, When Finance Intelligence Agent can help you close your books, you know, 10 days faster than you ever could before, which it can, you know, jump on that, right? Be ready to move in any direction, but only adopt AI that has real value for your business. All right. So, Jeremiah, we've covered a lot in today's conversation from what's new with Sage Co-Pilot, what's being announced here. We've talked about risks and guardrails and everything in between. But as we wrap, what is your one most important piece of advice, right? Getting back to trust and good enough, maybe not being good enough. What is the big takeaway to make sure that people can have AI that they can feel confident in and they can trust those outputs? Yeah. Oh, I just want to share that people shouldn't shouldn't give up on this. I think there's a lot of skepticism around AI that's arisen from, you know, overinflated promises. But I think, you know, all the AI we build at Sage is built with grounding and with access to real answers and real data. So if you've seen it happen, you know, something else hallucinate, that's not good enough. You can maintain a really high bar there. There is AI that you can trust out there in the world. It's being built by Sage. And what does that really mean? It means that you're confident in its answers. As a human, you control what's going on. And that AI is accountable and auditable for what it's done inside your finance system. All right. What an amazing conversation. So we covered a lot in today's show, but there's going to be a lot more. So make sure to check out today's daily newsletter for everything else that happened at Sage. Jeremiah, thank you so much for taking time to join the Everyday AI show. We really appreciate it. Thank you, Jordan. All right. So if you miss anything, like I said, make sure you go to youreverydayai.com. We're going to be recapping everything that happened at Sage Future and a lot more. Thanks for tuning in. Hope to see you back tomorrow and every day for more Everyday AI. Thanks, y'all. Every AI vendor in your inbox is promising to revolutionize finance. Most of them have never set foot inside of a controller's office. Sage has. At Sage Future, they double down on AI you can trust for finance. AI that gives your team confidence with results they can explain and verify. Control so people stay in charge of the decisions that matter. And accountability so every action can be traced. Plus, a faster way off legacy desktop tools and into the cloud alongside AWS. That's how you actually adopt AI in mission-critical work. Check out sage.com for more. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more AI magic, visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.