A new review shows VA clinicians are using AI tools for documentation and decision support, but there's no system in place to detect mistakes or respond to risks. Without better oversight, the technology could unintentionally affect patient diagnosis and care. Here to explain the warning and what needs to happen next is the VA's Inspector General, Cheryl Mason. Ms. Mason, it's lovely to have you on the show. Thank you for having me, Terry. You've just published a really important notice at the VA. What prompted you in the Office of the Inspector General to take an early look at how the Veterans Health Administration is using generative AI? Well, Terry, we've been looking, the OIG has been looking at AI in the department since late 2023. And this includes AI's chat tools for both clinical care and documentation. So this wasn't prompted by allegations or complaints. This is just part of our normal oversight that we are charged to do under the statute. And so what we really wanted to do was make sure we understood how and in what clinical settings these two tools are being used, right? So again, it goes to our oversight, understanding the environment the clinicians are working in and other staff and how they use these tools to support and enhance their care for veterans. And how in this case, are they using the chat AI tools to support clinical diagnosis? So it looks like from our review is they are reviewing it, you know, just using the chat tools in general to apply to cases. I mean, how they use it specifically is a little tricky because they all use it differently, which is part of our concern. But the big concern here is however they're using it in whatever manner, the concern is that AI hallucinates. There's research done on this. There's been many, many illegal cases on this. And so when you use an AI chat in clinical documentation, if the process is hallucinating, that can have an impact on patient diagnosis and management. And that's why it's so important to have the person involved, which according to the department, their recent press release, that they are doing that. You made this announcement as a preliminary advisory. Why did you take that step instead of waiting for the whole report and assessment to come out? So our PRAMs, our preliminary advisory memos, are really put out in a way when we have concerns that there is something happening or a situation we're seeing, whether it's in the inspection part of the house or in the audit part of the house that we need to get out quickly. And so the concern here is we didn't issue recommendations in it, but we see the need for NCPS and VA to communicate and coordinate. And so we really want the USH to take a look, the Undersecretary for Health, to really take a look and really evaluate the need for integration of AI-related risk monitoring. This goes, Really, our concern here goes to how they're qualifying the risk monitoring of using AI and if they're categorizing it the right way. It should be in our viewpoint. We recommend that it should be listed as a high impact issue when there's challenges or issues with the AI and chat tools. And that would raise its level of paying attention to it, put it at higher risk. And currently, the VHA and VA do not have it categorized as a high impact. And so our concern is there's a lack of control or oversight here on the AI tools, the chat tools particularly. One of the acronyms that you included in that last response was NCPS, which is the National Center for Patient Safety. Talk to me about who that is inside the VA architecture and what role they should be playing here as AI might roll out into clinician practice? So patient safety is extremely important in the department. They really, the patient safety experts track, they look at trends, they can intervene and alert on the situations, they can educate the staff on risks. And so they are, these are foundational functions that the patient safety program has to look at different ways of operating around patient safety. And that's where our concerns come in is because patient safety really isn't, these are not listed as high impact issues when there's chat problems, the AI chat issues or safety issues. They're really not tracked as an AI tool is not being tracked as a patient safety issue. I'm speaking with Cheryl Mason. She's the inspector general for the Department of Veterans Affairs. So with that as background, let's go back into sort of an organizational assessment here because VA has been standing up new AI policies over the past year and yet what you found here indicates that perhaps not everybody who ought to be at the table has been at the table especially when it regards deploying AI in cases of patient safety Are there new guardrails that you think are necessary or new processes to make sure that VA really takes a comprehensive approach as it's putting new AI policies into place? You know, that's a great question, Terri. I really think the biggest issue here is how it's categorized. Classifying the tool the right way in the VHA world will then increase the level of oversight and tracking on it. And that will be able to ensure that the near misses, if there are any, for patient safety purposes can be categorized and tracked with the patient safety program. And, you know, the VA is changing very quickly, as you noted, using a lot of different tools. And we are finding that in some cases they do have the right people at the table. It's just in some of the things that have been operating for a while, they're going back to regroup. And that's why this PRAM was issued, this preliminary report was issued to try to get the word out that this was another area they need to take a look at. And who is the they here when it comes to properly classifying these kinds of AI tools? Well, it would sit in two places, actually three. The top, of course, would be the secretary, but then the undersecretary for health, as well as the CIO, the assistant secretary for IT. Currently, that is the deputy, the deputy secretary who's doing those duties. So those three people would have really the primary duty, the Undersecretary for Health with the patient safety issues, and then the IT office, the OIT office, Office of Information Technology team under the Deputy Secretary should also be looking at it from their side. So it's a dual tracking situation. Does this kind of preliminary advisory get out to the clinicians and doctors now so that while the department is thinking about what policies or processes it wants to put into place, the practitioners know about these risks and can do something about them? Yes, exactly. And thank you for reminding me about that. That's one of the issues with the PRAMs and why we're issuing them, because we can get those out to the clinicians. This goes to the undersecretary, but then it is publicly released to the public as well as internally in the department. And just as recently as January, I was actually visiting in the field. And one of the facilities I was visiting noted that one of our previous prams was very helpful in getting information to them so they could react to it and think about it differently. So yes, this also gets the information out to the field quickly. We don't have to wait for the full report. We can give you a heads up, an alert, say, hey, pay attention to this, be aware that, you know, we might need to pay a bit more attention to the helpful tool that AI can be that makes sure there's human engagement. And I know while we're waiting for the report and the official recommendations, they'll come out later. But what happens now inside VA? What do you want the key decision makers to be thinking about? What changes in immediate actions should they be putting in place while your oversight continues? Well, really, the focus is for the Undersecretary for Health to really take on that conversation and coordination with the patient safety team just to make sure that we're looking for the integration of the AI-related risk monitoring into the existing patient safety programs they have. They already have great patient safety programs for risk. Just adding this one to the list would be a recommendation or a suggestion that we have. Again, we didn't make recommendations in the PRAM, but it is a suggestion that we would like them to take a look at. And that's something that when I speak with the secretary and other leaders in the VA, the Undersecretary for Health. Those are things we talk about and why we are issuing this. So, you know, that's part of our job. Give you a heads up and then hopefully you react. So this isn't a stop sign when it comes to AI deployment, but sort of a flashing yellow, making sure that they're doing some additional thinking about process and rules. That is well put, Terry. That's exactly what it is. It's a flashing yellow. It's a caution. AI can be a great tool and should be used as a tool. And that is what we're advising the department. We would like to make sure you can use the tool with that flashing yellow light to put in some guardrails around it, make sure it's being tracked, make sure it's being classified the right way as a high risk item as appropriate, if appropriate, and then use it with human engagement. I've been speaking with Cheryl Mason. She's the inspector general for the Department of Veterans Affairs. Really appreciate you sharing this important finding with us. Thank you, Terry. I really appreciate your time and thanks for having me on to talk about this. We'll post this interview at federalnewsnetwork.com slash Federal Drive. Subscribe to The Federal Drive wherever you get your podcasts. Sean O former Navy Secretary NASA Administrator and Senior Leader across multiple presidential administrations reflects on the principles that shaped his long federal career In this episode of Lessons in Leadership he joins Shane Canfield, CEO of WEPA, to discuss motivating teams through uncertainty, leading across agencies, and staying rooted in mission during times of public scrutiny and change. Hello, and welcome to the Lessons in Leadership podcast. I'm your host, Shane Canfield, CEO of WEPA. Today I'm honored to be joined by Sean O'Keefe, former NASA Administrator, former Deputy Director of the Office of Management and Budget, former Secretary of the Navy, former Comptroller and CFO of the Defense Department, and Professor Emeritus at Syracuse University's Maxwell School of Public Affairs. Welcome. Welcome and thank you for joining. Well, thank you, Shane. Delighted to be with you. Appreciate it very much. What first drew you into public service, and when was it evident to you that leadership was an important part of the roles you played? Well, I grew up around the public service. My dad was a career naval officer. He was a submariner, and we lived all over the place throughout the country. And as a result, I got an up-close and personal understanding through him as well as through all the folks he associated with and the environments he was working in, just what the extraordinary value is of public service and what is it that would motivate people to want to be engaged in it and to be really involving themselves in what is something larger than themselves. And the only place that that is really formulated properly is within the public sector and the public service that, again, is the beneficiary or the beneficiaries are the citizens of the nation, the community, and the region in which you live. Did you think about leadership early on, saying, I need to be a great leader, and therefore I'm going to learn how to do it? Or did you just naturally evolve into it? Any thoughts on that? I guess it evolved by the great good fortune of being a witness to several really extraordinary leaders, people who really did take the initiative to engage others in the task, to pull together and motivate folks to really contribute their very best to achieve an outcome. and I saw the value of that on multiple occasions and found that was the primary ingredient, the trait, the characteristic that needed to be exhibited is the fundamentals of leadership in moving everyone engaged towards a common objective. You have served under several presidents. Can you talk a little bit about that? Were there challenges to overcome or were you away from politics so you didn't get... Talk about that a little bit. How does it differ as you serve through a transition or does leadership stay the same? Oh, no. It's really critically important in transition cases, to be sure. But the two presidents that I served in their administrations were President George H.W. Bush and President George W. Bush in two different eras, obviously. But as a consequence, that was the closest I came to understanding the nature of the individuals who were in that leadership, ultimate leadership capacity, and their capability to motivate. Both of them were just extraordinarily inspiring people who motivated everybody to be part of the solution. But most importantly, they also had a capacity of really designating what something needed to be done. And you knew that they were perfectly prepared to do it themselves. And that made the task that much more achievable. There were several different circumstances in understanding what the objectives of the senior leadership are all about. You've worked for different kinds of organizations, political administrations, and then government departments, agencies, and then private sector, and now academia. How does that affect your leadership style? Or do you find that leadership, the core tenets of leadership are applicable no matter what the organization you find yourself in and having to lead? It doesn't matter where you are, that the core leadership characteristics are the same? Are they different? Are you adapting? Well there a common myth in my judgment and I probably the minority view of this in the academic field of public management That common myth is that you can apply the same principles in the public sector that you do in the private sector I never found that to be an impediment. It was always a circumstance where strategy, the focus on the talent, people engaged, the process to going through decision making, the integrity imperative that must be there in order to demonstrate the absolute conviction of what's intended here, as well as the moral standing of what it is you're attempting to achieve. All those factors are very much the same. It's going to have its nuances. and there are very distinctive differentiators within those organizations and those sectors. But at the same time, those basic principles, those tenets, foundations of what make up organizations and organization theory, in so many ways are fundamentally the same. And how you employ those may be slightly nuanced to match the condition, but they all are imperative in the same direction. A little bit of a different direction. You were part of the very first cohort of what is now Presidential Management Intern Program, later became the Presidential Management Fellows Program. What did that early experience teach you about leading within government? The presidential management intern program as it started was designed for just that purpose, a focus on public management, a focus on implementation. And its evolution over time has been extraordinary. It mystifies me why that has been the subject now of an executive order that has dismantled it. And it's absolutely unconscionable, given the nature of the hundreds, thousands of people who've moved through that program in the course of the last nearly 50 years that have gone on in some cases to be senior executives within the federal government. And in other cases, they've moved on to other kinds of pursuits. But even with that, even for those who have left the service, they walked away with an appreciation of what public service is all about, why it matters, and why this is so critical to get it right. Because by and large, the reason why these functions are delegated to the public sector is because no one else can do it. Usually that's the case. It takes that kind of focus of a public entity in order to achieve an outcome that frankly would not be achieved in other spheres. So as a result of that, it gave me an incredibly in-depth understanding of that process at a very early stage in my career that has lasted the entirety of that professional experience. A closing question. Who has influenced your leadership style? And what about their example stays with you? And that question can be cast broadly. So, you know, inside government, outside government, perhaps not government at all, somebody that you've met, reported to, and mentored by, a broad net. But when you think about broadly, when you think about who has influenced you, who comes to mind? I would have to say that the number one, hands down, greatest influence during the course of my life was my dad. He was an engineer, so everything was a calculated risk. I think I was probably 25 before I realized the answer was not to be found for everything in a slide rule, but that was his bent and his mindset. And he taught me how to think critically. And it really was one of the most extraordinary attributes. And he was an inspiring parent and mentor in so many ways. All the different things I've done throughout the course of his life, I had the opportunity through much of it to be able to consult with him on a variety of things and always walked away with a more informed perspective. Ed was an extraordinary public servant in his own right. Well, Sean, thank you very much for sharing your insights and experiences. It has been a pleasure talking. I'm delighted to be with you, Shane. Thank you so much for doing this. And this is Shane Canfield, CEO of WEPA, reminding you to empower your team and embrace challenges. Until the next time on Lessons in Leadership. Find the full podcast and future episodes of Lessons in Leadership on the Federal News Network app and anywhere you enjoy your podcasts.
