Embracing Digital Transformation

#337 Mastering Vendor Security in Financial Services: A 12-Year Journey

29 min
Mar 25, 202625 days ago
Listen to Episode
Summary

CIO Galen Councilman of PenAir Credit Union discusses the complexity of managing 25+ vendors in financial services digital banking, emphasizing vendor security risk assessments, integration challenges, and the critical role of synchronous communication in resolving multi-vendor issues.

Insights
  • Financial services require fundamentally different vendor ecosystems than healthcare due to digital-first customer expectations, necessitating complex multi-vendor integrations rather than single-platform solutions
  • Risk assessment governance must involve board-level decision-making on medium-risk acceptance rather than individual IT leader approval, distributing accountability across the organization
  • Synchronous vendor collaboration (all parties on one call) resolves multi-vendor issues 30 minutes versus 2+ weeks with asynchronous ticketing, indicating process design significantly impacts operational efficiency
  • Lack of standardized data models across credit unions forces custom ETL development for each vendor integration, creating substantial technical debt and maintenance burden
  • AI adoption in financial services currently limited to data movement automation via Python rather than integration orchestration, suggesting immature tooling for vendor ecosystem management
Trends
Digital-first banking is table stakes, driving exponential growth in vendor ecosystems for credit unions and smaller financial institutionsRegulatory scrutiny of vendor security (NCUA annual audits, quarterly risk reporting) is increasing, shifting security from IT-only to board-level governanceSynchronous communication patterns emerging as critical operational practice for multi-vendor incident resolution in complex ecosystemsCustom data transformation pipelines becoming standard operational burden for mid-sized financial institutions lacking industry data standardsAI-assisted code generation for data integration showing early adoption but not yet addressing orchestration or vendor management complexityPrivacy and security divergence in financial services regulation creating distinct compliance requirements compared to healthcare sectorCustomizable core banking platforms creating fragmentation across credit union industry, preventing code/process sharing and standardization
Topics
Vendor Risk Assessment and NIST 800 ComplianceMulti-Vendor Integration Architecture in BankingDigital Banking Platform Vendor EcosystemsEncryption Certificate Management and Root CA UpdatesData Transformation and ETL Automation with PythonNCUA Regulatory Compliance and Board ReportingSynchronous vs Asynchronous Incident ResolutionCredit Union Core Banking System CustomizationAI-Assisted Code Generation for Data MovementInformation Security Risk Appetite StatementsThird-Party Vendor Security AuditsRemote Deposit Capture IntegrationOnline Banking Platform ArchitectureHealthcare vs Financial Services Security ModelsVendor Contract Security Requirements
Companies
PenAir Credit Union
Guest's employer; case study for managing 25+ vendors in digital banking with NCUA regulatory oversight
Microsoft
Infrastructure provider; PenAir standardizes on Microsoft Office and Outlook for productivity
Cisco
Network equipment vendor; NCUA examiners reviewed Cisco router configuration logs line-by-line during audits
National Credit Union Administration (NCUA)
Regulatory body; conducts annual audits with 15-20 person teams examining vendor security and risk assessments
People
Galen Councilman
Guest discussing 11-year tenure managing vendor security, 26+ years IT experience from healthcare to financial services
Dr. Darren
Podcast host; Chief Enterprise Architect interviewing Councilman on vendor risk management in financial services
Quotes
"That takes over 25 vendors just to make that happen. So when one of our members downloads our app and they sign in, they're going to potentially interact with over 25 vendors to make that whole experience work."
Galen CouncilmanEarly in episode
"Privacy and security are two different things. They're related, I believe, you know, in my opinion, they're very related, but they are two very different things. You can be private, but not necessarily be secure."
Galen CouncilmanMid-episode
"We've always found that anytime we can get all of the vendors on a call at the same time, which they never want to do... 30 minutes and then it's done. It's taken us two weeks to get there."
Galen CouncilmanMid-episode
"If it's a medium or higher, we either have to mitigate it... or in some cases, we've had to accept some medium risk. We've never accepted anything that's high or higher than that."
Galen CouncilmanLate episode
"You're living in the trenches, not this big esoteric strategy, you know, no, you're living this every day."
Dr. DarrenClosing remarks
Full Transcript
That takes over 25 vendors just to make that happen. So when one of our members downloads our app and they sign in, they're going to potentially interact with over 25 vendors to make that whole experience work. Welcome to Embracing Digital Transformation, where we explore how people process policy and technology drive effective change. This is Dr. Darren, Chief Enterprise Architect, Educator, Author, and most importantly, your host. On this episode, vendor risk in financial services, cybersecurity and AI, with special guest, Gowlin Councilman, CIO of Pin Air Credit Union. Galen, welcome to the show. I'm glad to be here. Hey, this is really interesting when we talked, where was it last week, a couple of weeks ago that we originally talked? Yeah. Couple of weeks ago. Yeah, this could be a really interesting angle that I haven't had on the show before. So I'm like, oh, this could be really cool. But before we dive into managing vendors and AI, and this whole new realm that we have, Galen, everyone that listens to my show knows that I only have superheroes on the show, and every superhero has a background story. So Galen, what's your background story? What's your origin story? All right, well, I am not a superhero, so let's start there. All right, so cute Milady is one of your superpowers. Okay, that's a good superpower for a superhero. Yeah, I meant earlier to say, hey, I'm glad to be here. Thank you for having me. So I have been in IT going on, geez, 26, 27 years now. Started off working at a local computer shop back in like 97 timeframe, back going if you can imagine a time when we had local computer shops, you know? We had a local place that I started and kind of cut my teeth on learning how to build computers and service computers and all of that. And I'd grown up loving technology. I'd always, I remember when my father brought home our first personal computer, it was an IBM clone back in the day, an 8088 processor. There you go. With MS-DOS on it, I remember using XTree Pro for like organization and everything, and I loved this, you know? And I would tinker around bulletin board systems and I would do a little bit of minor programming and things like that. And I remember tearing up my dad's computer one day, doing some file structure stuff and accidentally deleted the entire, that was a back, there was no protection, you could just delete the whole operating system by accident and then when you reboot it, it just, you know, nothing happens. And so that's where I started. I always tinkered around with computers and then started at a local computer organization, local computer shop, and then from there, got a job at a architectural engineering firm as a network support technician. So, you know, running around doing everything from desktop support all the way down to, that was when we had compact servers that were, you know, desktop, like standalone servers, sitting next to a telecom rack at all the branches where this company was and moved into healthcare. That would be probably the first big start of my career at a larger organization of running IT for a bigger firm. Started there as tech support and then moved my way up into network engineering. Back in the day, I used to program Cisco routers and switches and firewalls and I was certified in Microsoft Exchange and had all the Microsoft certs and so I did all of that, you know. So you're a true hands-on IT guy, right? You've done it all. You've done all the hands-on type of stuff. I grew up in that. Yeah, so, but you're not that anymore. Correct. So from that time in healthcare, you know, I've always just operated by that concept of if I can be trusted with little, maybe I can be trusted with more and I've just focused on doing the best that I could at the job that I had at that time and then more things opened up to me. And I moved into running the network operations for that healthcare organization. So had grown a passion around HIPAA security and the high tech role and had during that time built up a passion around information security in that time in healthcare and then started running the network operations team and then from there became the director of IT. And so by the time I left healthcare, I spent 12 years of that organization. When I left there, I was running the entire IT shop and I could be different things to different organizations for there, that was everything from tech support, network operations to programming, software development, data and analytics and the PMO that was there. And I've really, really enjoyed that. So I left and then moved over into the financial industry where I landed at Penn Air. I've been here right at three years now. So just hit three or not three years, sorry. Just hit 11 years now. Sorry, I don't know why I said three. I was thinking March 3rd for some reason. So yeah, I've been here 11 years and started off as information security officer. If any of you who've been in management, so when I left healthcare, I was looking for either to continue my career either in information security or in IT leadership and just happened to find a security job first and landed here as a security officer. And actually I liked that. I liked not having the responsibility and the weight of managing large teams of people and all the headaches that come along with the people management side of things of leadership. But then- I do know, that's why I'm- That's why I'm not managing people. That's right. It's not for the faint of heart, right? It's not for the weak. It's a very, you know, a lot of responsibility in that. So. ["Safari Engineer"] Well, and just like you, I'm a technologist, I'm a software engineer and I tinkered around. I like deterministic problems to solve. Computers behave the same way every time. People do not. Yes. People are harder to work with than computers for technology guys like you and I, right? Yes. So. Clearly empathize with what you're saying and agree. But there's more joy. There's more joy out of interacting with people and helping people. It's just gotta put that extra effort in, I guess, is the right word. So let's talk a little bit about, you know, you're in financial, you work for PNR, it's a credit union. Very, was there a big difference moving to financial from healthcare? Or were a lot of the same IT and specifically cyber security? Was there a lot that transferred over or was it a completely different way of thinking? A lot transferred over. I mean, you think through things like infrastructure. All of that was very transferable. You know, you still need the same routing, switching firewall, cloud technology, productivity type operations. You know, do you have a CRM? Do you have, in addition to productivity, things like CRM, ERP systems, you know, you have all of those things that cross over. What I found was different though, was my time in healthcare, very hyper focused on privacy, but not as much on the security with the organization that I was at for 12 years. So everything was around HIPAA, HIPAA rule and privacy, keep in patient information private, but not secure. I never in 12 years had an actual IT security audit at that healthcare organization. Lots of privacy audits. How do we have our systems set up and what were our policies, procedures to keep information private? But privacy and security are two different things. They're related, I believe, you know, in my opinion, they're very related, but they are two very different things. You can be private, but not necessarily be secure, you know. And when I came over to financial, it was completely, I mean, we are audited all the time. So we have to pay for third party audits on a quarterly basis, external, you know, pin tests and all of these sort of things, just kind of things that we didn't have to do at the organization I was at in healthcare and we're also regulated by the NCOA, National Credit Union Association, that they come in annually and do a big, they bring a team of people here across our organization, like 15 to 20 people. And I usually get anywhere from one to three examiners that come in and they're deep diving into what we do. I even at one point had examiners, some would say they may have, we're going a little too deep with us, but they were actually asking for Cisco router configuration logs and we're going through line by line, the running configuration of some of our operating equipment. Do you think that reason for that is because you're dealing with people's money? Absolutely. It's a bigger target. I mean, data, I mean, people's patient data is a target, but the payoff is much faster, I guess, with if I get into a credit unit, I have free reign of the thing, I can do whatever I want, different, because with patient information, I have to do a ransomware attack or something else. And so that kind of makes sense to me. Yeah, it's where the money's at, right? So if we can get that, that's their main target, like you said, that's what they're after. Yeah, with healthcare, if it's identity theft, I mean, money is always the ultimate goal somewhere along that chain, right? Yeah, that seems like, yeah. But it's several steps up or away from that actual payday, whereas if you can get into the money, yeah, and we're talking people's financial assets, we're talking about their stability and their future here, if their bank accounts get drained, that's a very big deal. Yeah, absolutely. So have you found, because when we first did our intake on this, we talked about managing your vendors, have you found the ecosystem of vendors really different between financial and healthcare, or are there a lot of commonality between them as far as your suppliers of software solutions or hardware or things like that, or is it completely different ecosystems? I find it different. So where I came from with healthcare, you really had your core applications, like your medical record system, you would have great planes or whatever your accounting and ERP package was. And in financial, it's similar, you'll have that core banking application, but the challenge that we have, like in healthcare, we did not, we weren't offering out digital services and now as a financial institution, if you're not 100% digital, I mean, it's table stakes these days, right? It's like we all expect to do everything from our phones. And so to make something like that happen, there's not one silver bullet out there, there's not one vendor that just does it all soup to nuts and does it very well or to the, that meets all of our needs as a credit union. So that's one of the biggest challenges there is finding all of those different vendors and having them all play together so that you can have the digital offerings and digital channels that your members expect. Also as an organization, we push for excellence. So we try and be the best that we can be in everything that we offer. And so finding that and getting all the right vendors and players is definitely a huge challenge. The underlying infrastructure, kind of similar. We're a Microsoft shop, you know? So everybody uses office and outlook and all of that kind of infrastructure side of it. But it's really what we're offering to our members. So that's a big challenge. So do you find the hardest part of working with all these vendors is security, a hard part or integration or contract and Cush? I mean, what are some of the things that you run into that are so different from healthcare? Healthcare, pretty small ecosystem, frankly. Right. I would say integrations is one of the biggest pieces that's a challenge because, you know, let's take a bank, especially a larger bank. They'll have the resources to hire teams of software developers to create their online platform so they can design and build exactly what they want and how they want it. I can't do that. I'm not at that resource size. So we have to rely on bringing in different vendors. So if you take something like what we offer for our online banking platform, that takes over 25 vendors just to make that happen. So when one of our members downloads our app and they sign in, they're going to potentially interact with over 25 vendors to make that whole experience work. You know, so you have a main platform. So that's one vendor, you know, so that's kind of like what you mainly touch and feel most of the time. But when you switch over to, let's say, remote deposit capture where you need to take a picture of a check and deposit it, that's a whole nother vendor. But all of it needs to look seamless, you know, so it doesn't feel jarring. So when a member clicks on that, it doesn't feel like it's jumping over to another application or platform or another vendor or login or all this. So it all feels completely integrated and seamless. But that's, it's a lot, I'm going to say it's a lot easier. I think it would be a lot easier. If I had an army of software developers could just design it how we want it. I think that would be a lot easier than just going out and finding all of these vendors. And then you have to plug them in and then make sure that the integration works and make sure that it's seamless and streamlined. And it all feels like it's the same application. It's definitely a challenge. And so that's one example, you know, but I could rattle off a ton of them that we have that all integrate, you know. Yeah, I'm sure. How have you found that maybe AI can help out with this? I mean, the AI models are getting really interesting in that they can do these integrations kind of, it's very simple and you can do some front end vibe coding on the front end and connectivity in the back. Have you guys experimented with any of these sorts of things to help increase that friction? No, as far as we've gotten with AI is just helping us with the data movement side of things. So if we're, you know, we use a lot of Python here to move data between vendors and to make some of these integrations happen. So we've leveraged that to create code that we can then automate. And in some cases have automated, you know, but the other challenge and what I was gonna mention about this too is when something breaks, that's always a big deal, you know. And it's being an IT and coming from a technology background, I'm sure you can relate to this when people love pointing fingers. Oh yeah. You know, we point the finger, they point the finger and then it's just, you know, you got a three-way everybody's, you know, everybody's pointing the finger. And so it's, we've always found that anytime we can get all of the vendors on a call at the same time, which they never want to do. Everybody wants to handle through tickets and phone, you know, separate emails and it ping-pongs back and forth and then things draw out. And then I get our board of directors or putting pressure on my CEO and myself, when is this gonna be up? What is this gonna be up? And it's like, guys, we gotta get them on the phone, we gotta get everybody on a call now. And then when we do that, it gets fixed, you know, it's magic, it happens. It's amazing. 30 minutes and then it's done. It's taken us two weeks to get there, you know. We have a phone call and we get it fixed in 30 minutes. So is that the best practice then to do sync? I call that like synchronous meetings, right? I mean, everyone's in the room at the same time instead of this asynchronous, there's a lot of miscommunication, I guess is right. Or mischews on communication when you're doing everything asynchronous. I think you've identified here. Yeah, yeah, I mean, because one instance I'm thinking of is we had an issue with between three vendors. One of the vendors had changed a encryption certificate in the background and tell anybody. It shouldn't have affected anything because it was a root CA change. And okay, it wasn't an issue. And so, everyone was pointing the finger. Well, it's not us, it's someone you're in. You're not passing the right encryption and all of this. And we get them all on the phone and when they can see the real-time logs, okay, try it now, click. Oh, and that's when it's like, oh, we didn't give you the new root CA here. And they, you know, they send us the, what, the pks file or what I can't remember at this point. But you know, they send us the group of certificates, and we get those loaded in the web server and then everything was fixed, you know, and we were pulling our hair out for over a week on that particular issue. How do you handle the security handouts? Because you mentioned these, the certifications and things like this. When you have so many vendors and you're actually moving data between all these vendors, how there's, there's so many questions on this one. But let's talk about security first. How do you handle the security between these? Do you have a common security design pattern that you use? Because that's a lot of vendors, 25 vendors, that's a lot. It's a lot. Yeah, for sure. So we rigorous risk, risk, risk assessments. And that's the number one thing that the issue is always looking for us in our regulation is the credit union performing reasonable information security risk assessments of all vendors and everything that we do. So when I first got here, we put into play a, just adhering to the NIST standards, the 800-document on guide to risk assessments. So we follow that and do that every, with every vendor, every change that happens, we go through that. And the purpose for that, that helps us dig up rocks and know where do we need to dig deeper into some things and either say that a vendor is not a good fit for us or dig a little bit more into that vendor to understand, okay, here's some things we need you to add to your contract. Or here's some things that we need you to do differently with this implementation to make sure that it meets our security standards. So we do those risk assessments. Those actually get reported up through our board of directors. So per NCUA, we've got to report that to the board, make an annual information security program presentation to them and all of those risk assessments go to them. We actually do that quarterly instead of just annually where we vote on risk. Well, can you explain that a little bit more? Because what I'm hearing, and I know the answer to this, but I don't think my audience understands, when you talk about risk assessment, shouldn't you always do like zero risk? Sure. Should there always be zero risk? Yeah. What's the deal? Let me just turn off the internet and then we have no risk at that point, right? But how do we operate as a business at that point, right? Right, so there's always some risk. So how do those meetings go? Is there a calculated risk? And you're saying, hey, this is, we're willing to connect to the internet or give our customers ability to connect to the internet, even though there's a risk, we just have to understand the risk and the mitigations. Why even take the risk in the first place? That's correct. So what we do as an organization, what the NC way asked for is that we have a risk appetite statement. So part of our information security program, our board, along with management, have agreed upon our risk appetite is low. So when we do a risk assessment, so we're looking at threats, vulnerabilities, mitigations, and then what is that residual risk with those mitigations that are in place? If it's a low or if it's a low or less than that, so low or very low, then we accept that. If it's medium, we don't. So we have two options here. So if anything is a medium or higher, we either have to mitigate it. Let's say it's a vendor in this case, new vendor coming in, we have a medium risk here. That vendor is either gonna have to make changes that satisfy that risk for us to bring it down to a low, whatever it is, say an authentication issue or something when how they store data or process data. AI is an example. Are you using our data to train models along with other clients of yours? That's a big no for us. So they would have to give us guarantees that they would not do that, otherwise that's gonna remain a medium. So what we would do there is that particular vendor, we're not gonna be able to do that unless we can mitigate it. Or in some cases, we've had to accept some medium risk. We've never accepted anything that's high or higher than that. But sometimes we've had some medium type risks that were around some things that could not be mitigated, but the scope was much smaller with this particular organization. And so we voted as management and with the board to accept a risk. But that happens a lot less than us mitigating it. No, so I like how you describe that. It's, and I love how you have your board decide that. So it's not just, you know, Gowen said it was okay. That would put a lot of pressure on you as an individual saying, hey, I'm willing to take a medium risk because of this. Instead, you're hearing from lots of different people and you're governing board for this. I think that's pretty clever. We'll make the decision together, right? Yeah. I believe that helps. All right, so my next question has to do with data. So obviously if you got all these vendors, is there a common data model that you're using or that exists, is there a standard that exists out there for all these? Or you end up writing all of these data transfers because you kind of hinted towards that a little bit that, hey, I'm converting data from this format to that format. This field means this there. Is that something you guys have to do with these integrations? Everyone's different. Everyone's different. Yeah, we don't have, there's not a common data model. Oh man, what a pain. There's nothing like from healthcare, I left there, I had a HL7. You know, there was a common data format for us to share information back and forth. Here there is not. There have been organizations that have tried to design common data models for credit unions and they just never have worked because we're all unique enough in the members that we serve, the communities that we serve, that our data, we need data differently than each other, you know? And so everything's, yeah, it's all a challenge to figure out how do we get that data, transform it and then bring it into our systems and make it usable for what we need. So yeah, so you go back to those 25 vendors, well, that's a lot of different data. And so it's a lot of work cleaning that up and getting that brought into, in our case, in our warehouse, you know? It's, yeah, it's definitely a big challenge. We also, our core that we have is known to be very customizable, which is also a weakness as well as a huge benefit because- As a strength, yeah, as a strength and a weakness at the same time, yeah. Yeah, but it means all of us that have that same core application, none of us use it the exact same way because we've all tweaked it and modified it and done our own customizations. And so we can't even share things. So sometimes we can share code back and forth, but there's a lot of, you know, we have to tweak and modify things to fit our environment versus, versus theirs, yeah. And it's much more complex than I think most people realize. Yeah. I think it is. I've never managed anything like it before. It's, it's intense. It sounds pretty intense, especially considering, I mean, how big your staff is. I mean, your staff can't be massive, right? Like you were mentioning the big banks, they've got big, huge teams that do all this stuff where you've got a smaller team putting it all together. It must take a lot of discipline. Yes, sir, sure does. So never a dull moment. Yeah, I bet, I bet not. So Galen, if people want to find out more about Pinair and what you guys do, and maybe learn more about the best practices that you guys have, you know, started there, how do they go about doing that? Pinair.org, great starting place. You can always find me on LinkedIn, Galen Councilman. Happy to connect and share more and chat with anybody. Hey, Galen, this has been great because I'm talking to someone that's living in the trenches, not this big esoteric strategy, you know, no, you're living this every day. So thanks for coming on the show and sharing. Very welcome. Thank you for having me. Thanks for listening to Embracing Digital Transformation. If you enjoyed today's conversation, give us five stars on your favorite podcasting app or on YouTube. It really helps others discover the show. If you want to go deeper, join our exclusive community at patreon.com slash Embracing Digital, where we share bonus content and you can always connect with other changemakers like yourself. You can always find more resources at embracingdigital.org. Until next time, keep embracing the digital transformation.