And people would say, what's the most important thing? Billing. So all these accountants would get together and talk about how quickly invoicing had to come back up. The reality is, from a business impact analysis, okay, accounts receivable goes out a couple weeks. Big deal. What do we need to do? Accrued billing. Who's talking? Who's writing? Who's producing? Welcome to Embracing Digital Transformation, where we explore how people, process, policy, and technology drive effective change. This is Dr. Darren, Chief Enterprise Architect, educator, author, and most importantly, your host. On this episode, Disaster Recovery for Executives, Why Cloud and SaaS Are Not Enough, with disaster recovery expert and CEO at Different Dev, Tom May. Tom, welcome to the show. Hey, thanks for having me today. Hey, Tom, what we're going to talk about today is actually dear to my heart. I've been to companies that have gone through disasters and weren't prepared for business continuity, like a hurricane hitting. That's never something you want to prepare for, but you have to. And they weren't prepared, a total disaster. But before we talk about all of that, everyone knows on my show I only have superheroes on the show. and every superhero has their own background story their own origin story so tom what's your origin story my origin story is probably a little unique in that i believe i came to be an expert in this area through the power of no um i had been a generalist in it forever so a little bit of everything jack of all trades um i'd like to say master of many but not all right and so there was a small firm doing DR before DR was even a thing. They had a file level backup business and they approached me several times to work with them. And I kept telling them, no, like, oh, we're mostly Linux. And I was like, well, you know, I kind of know Linux, but more of a Windows guy. Oh, that's what we're moving into. And through a series of no's, I finally go to work there. And I realized something pretty quickly back in the days of Carbonite and file level backup. And this company had it there were great gaps they saw and being supporting in that space and that nobody needs a file they need to get quickbooks working back then it was the whole shebang and so we started muddling through cobbling dr systems before this was even a thing and i think i've just grown with it and now i think or at least i like to think that i'm an expert in that area so you're kind of you're almost like you grew up with dr when group when dr was growing up you were right there with it absolutely i went through the challenges of how do you get it off site how do you get it backed up how can you actually recover it and then it was oh no you know somebody deleted it and but they're supposed to be able to delete it but that wasn't me so yeah i've gone through so many times like say i'm the guy that was in the room when it went bad and learned from things we just didn't know about at the time so you're you're a previous diamond in the rough now you're pure diamond, right? Because you've got all the hard spots knocked off. You learn through going through it. Oh, absolutely. Yeah. Absolute hive knowledge here. Learned and benefited from other people's mistakes. That's the best way to learn. It's making your own. So let's look at this from the perspective of an executive, right? Let's say I got a mid-sized company because those are probably a mid-sized and small companies are probably the ones that are at highest risk of this sort of thing. Cause big corporations, they've got, they've got big DR and BC business continuity programs. They've got the money that they can do it right. But what about the mid-sized guy, the small guy as an executive, what do I plan for? Why do I plan? And you know, how do I get started? What do I do, Tom? Well, honestly, most organizations think they're protected or they know they're not protected and both are dangerous. So if you're not protected, obviously you've got to do something. And ultimately, I don't think you are going to have the specialty on your staff, your main IT person or people. They may be really great at things like I was a generalist. You know, they're knocking eight out of tens and everywhere. If it's a decathlon, man, they're winning because they're hitting the scores across the board. But with DR, it has to be more targeted. You need a 10 out of 10 in it all day long. So if you don't have anything, you've got to find someone to help you. This is not something you learn on your own. If you have something and you think you're OK, the honesty is you're not. This is an ever evolving need. It is organic because everything in your environment changes. Think about how many times someone gets a new phone. Well, how many times is there a new security patch? How many times does someone save a document? It's forever changing. So you have to have the mindset as an executive that unfortunately, we don't like that line item cost that says DR and IT services. It's forever in your budget. It is ever growing, ever evolving. And so that was my advice to any executive is it's not a one and done. You didn't put in a new operating system and you walk away for three years. This is every day being vigilant about it. So is there some point in that there's a return on investment you have to be worried about? I mean, there's a certain amount of risk that maybe you're willing to take for disaster. Because it could get very expensive, right, to do it right. Oh, absolutely. We're great at spending people's money in IT. Give me an open checkbook. I had a client that was like, well, you know, can you give us a price with this? And, you know, I joked, I was like, no, I'm going to spend all your money. Really, what you have to look at is business drives this. Okay. And we forget that. We let the IT people drive us into fear. Oh, everything you know is going to destroy. They're not wrong. But a business impact analysis is really, really important. And I'll give you a quick high level example of something. Let's say you work in law. I worked at a big law firm and people would say, what's the most important thing? Billing. So all these accountants would get together and talk about how quickly invoicing had to come back up. The reality is from a business impact analysis. Okay. Accounts receivable goes out a couple of weeks. Big deal. What do we need to do? Accrued billing. Who's talking, who's writing, who's producing revenue. So now you look at those systems and that's true of DR. What we think is I need it on now. Maybe really. So it starts with the business. The business drives this. Right. I mean, you've got to prioritize then and then put a time. It sounds to me like put a timeframe on it too. Like you said, accounts receivable, two weeks is okay. But, you know, accounting, you know, accrual, I want that in six minute increments, right? Mm-hmm. Exactly. All right. So, all right, that makes sense. So, that's a good place to start is looking at your business and saying time sensitive. Like, I'll give you a great example. When I was CIO, I said, hey, we've got an email. We were running our own exchange server and we were, it needed an upgrade. It absolutely needed an upgrade. It was cobbled together. and I said it's going to cost us a quarter of a million dollars to upgrade this thing. Three weeks later, it goes down. There wasn't enough money in the company's coffers that they were willing to offer me to get that up and running right away. Right? So time sensitive? Yeah. If your company runs on email, then having email up is critical, right? I think it comes down to two concepts when we talk about it from an executive And IT loves to use these acronyms And so I say them and then I going to tell you what they mean And they'll make sense. All right. So you have RTO, return to operation. How long can this system be shut off? I need it in five minutes or 10 minutes or two hours or three days. Right. So your email was a very low RTO, 15 minutes. Right. That accounting system, two weeks. I mean, we're not shooting for that goal, but we're prioritizing. If everything's on fire, nothing's on fire. Then we have recovery point objective, RPO. Basically, what point in time does the data come back? So I turned it on within 15 minutes. Yay. How much data did I lose? Those are the two factors. And so when you talk about cost and analysis, and we can run up the gamut on this, think about this. You could say, no, I need that email up with zero data loss and I need it to turn on in two minutes. There are systems that will do that. What if it was a million dollars? Suddenly you'll get sticker shock and go, you know what? Maybe 15 minutes isn't so bad if that's only $15,000. So again, your business is going to drive it. And what's the takeaway executives is this. You need to work with someone who helps you make a bad informed decision. In other words, you are aware of all options. you weigh your risk analysis you know what you're gambling when you go into it and when and if it happens you're prepared because you chose that you may not like it but you actively chose a situation it's a better place to be than a you know poorly informed decision exactly i just didn't know for 30 more you mean i could have yes i should have told you well this this sounds like this is going to take some time to put together a disaster recovery program. It takes time. It's not like something I could do in an afternoon. No. And when you work with somebody, they'll really help you slip stream it. When I get involved with clients, one of the things that I like to think about is we have to protect the now, even if it's not the best solution, we need a solution. Something is better than nothing. Something right away, whether it's. Exactly. Got it. Minimum viable product, as we like to say in those product roadmaps. right? Get something into production. At least you have something, you know, it's not ideal. Now you can roadmap it. Now you can ratchet it up. I have no backups, put a backup. Oh, but somebody could get in and delete them. You're right. But at least you have layer one covered. You have something. So some things can go in relatively quick for people. And that's what somebody should really plan at is it's a phased approach. It's a roadmap. You won't have every bell and whistle. If you watch CNBC, you won't have every tick box marked, but you know what? When people don't take action, they blink away three years. When they take small steps in a year, it's completed. It's kind of like, how do you eat an elephant? One bite at a time. So that's your approach. Your approach is to say, let's do it piece by piece, come up with our plan, walk through the plan, adjust the plan as priorities might change, or as new threats come up, right? Because there's new threats to disaster. Disaster recovery is not just a tornado hit my building. There's lots of different disasters, right, that I can talk about. In fact, that's kind of the rarity nowadays. I lost power. I had a tornado. Most people have some servers somewhere. Is it in the cloud? Is it in the colo? They're providing you power and uptime and grids and generators. You're not buying those things. You're not building it yourself. And if you are, stop, unless you're a Fortune 100. But go to somebody that does that. There's reasonable costs to that. And it's on. It's the human error. It's the malware. It's the security, like you're saying. Those are the ones that we're mitigating. So when you think of DR, it really goes hand in hand with security. They interlock. You cannot have security without DR. You cannot have DR without security. They are just like an infinity loop together. All right. So that's really interesting because you talked a little bit about cloud or Kolo. They handle some of your disaster. Well, they handle more business continuity than disaster recovery, right? They have backup generators. They have backup internet connection. They've got those things pretty much covered. But that's not the same thing as having your own disaster recovery business continuity plan, right? Because cloud service providers go down. People don't believe that, but they do. And they actually, their percentages are not great right now. No, it's denial. We're talking maybe only 99, 98% uptime? Well, you're talking denial, right? You look at having an architecture on system and go, I don't know all these things and I'd have to spend all this money. If I just write a check, it's absolutely taken care of. Taking care of for me. Rule number one. Exactly. Rule number one, zero trust. I don't trust them. They're my friend. They're my vendor. I'm not mean to them, but I'm a paid pessimist. I have to have another route. Even so far as, well, I'm in Amazon and I have multiple zones in Amazon. What if some random thing goes wrong in an Azure, an Amazon, whoever? And I'm not picking on them, but what if that one thing goes and then you're down? What priority do you have? You have to have a backup outside of that main production space. Backing up to yourself is no different than when I started in this business. You put your little backup tape in your server, it back up all the data, and the admin would stick it on top of the server and go home. Well, where's your backup when it burns down? What happens in that cloud? If the backup is in the same cloud, you're not really winning. You can have one and should have one there, but it's the tertiary that makes the difference outside of the environment, something else. And you have to plan on it. You know, I'm glad you brought that up because there was a there was a shipbuilder. I won't name who they were, but they're in Pescaloosa, Mississippi. And their backup, they had a business continuity disaster recovery plan. They had a backup data center five miles from their main one. So when the hurricane came, it wiped it all out. Yeah. Well, luckily they had tape backups further on. But, you know, it ended up being somewhat okay. It took them like four days to get back online. But, I mean, their shipyard was completely destroyed by the hurricane. So, I mean, some of the stuff they didn't have to worry about. for some time, but it goes to show you've got to be smart about that. Backups in the same cloud service provider, probably not a great idea. Think about your SaaS as well. Well, PeopleSoft, you know, PeopleSoft, everybody has them on-prem. Then everybody's like, they do it in the cloud and they just do it for us. And they say they do whatever. Where's your audit? Where's your test? How do you really know what they're doing? And you may think, oh, it's no big deal. You have to understand that right now in every business, your data is everything. If tomorrow you had access to absolutely no data, no matter what system you're using, would your business survive? You could survive downtime, but what if it never came back? That's the big challenge with these SaaS vendors. These are your Atlassians making Jira and whatnot. Love those products. Major software developers, if they lose that, they're done. There are products out there that will back that up outside of it, even if it's kludgy to come back. Again, remember it's minimum viable product. You have something. That shipbuilder would be happy if the data was two weeks old, a month old. If they had zero, how do they even call their customers to go, hey, we're back online or hey can we get this shipment Imagine you can call no one That the scariness It complete obliteration So all right So let talk about our reliance on these SaaS providers a little bit Because, yeah, there is this illusion of protection. And maybe sometimes we relegate that to them thinking, you know, we don't have to worry about it anymore. What you're saying is we do. It's still our, we have to remember, it's still our stuff and it's still important to us. So we've got to, we've got to do a backup. So is it easy for most of these SaaS providers? Do they all provide a way of backing up outside of their system? And so how do I go about learning about how to do that? They actually don't. Most of them don't. Some will. I'll use like Salesforce. There is a way in which you can extract some files and somebody is going to have to re-architect the environment to come up. Then there's third party tools. Veeam has them. I think SAS is sure has some. There's some other providers out there that you would subscribe to and hopefully use someone like me to architect it where, listen, it's not that much. Let's go in. Let's back it up and have it recoverable because we have to back up and we have to recover it. It's a two stage, you know, double headed sword here, if you will. So even if you have the backup, how do you get it back? Third party is really the way to go right now. I'm not really aware of many that make it very clean to be able to do that. So it is third party. I like to export all my data. Right. Like Salesforce's spreadsheets upon spreadsheets upon spreadsheets the last time I looked. And I'm dating myself at about three years at that mark because now there's a third party tool we use and we're just like hooking in. Just hook you in. Got it. It's hard to architect your solution with one tool. What I'm going to say is one tool will do most of it. Most of the pain points are those SaaS providers. It's finding which backup company software do I use? Do they have this product in the roadmap? Is it live? Oh, wait, somebody else has it. So, okay, I've got vendor number one. I love them, but now I need vendor number two. One day, maybe vendor one releases it and I roll it back into single pane of glass, one vendor architecture. but you're going to have to be a little bit robust going back to the business systems. You have to know the data that you touch and where it lives to be able to protect it. So executives, all your reports, head chiefs and directors, they need to know what systems do we use online or not? And how are we backing up? It comes up with every system we design, not just the newest flavor. DR is part of that conversation. Okay. So that's why it's a continuous thing. thing. You need someone that's sitting there saying, all right, we added this new tool, new data, who uses it? All that needs to go into a disaster recovery plan, right? It can't just be in someone's head, right? It's the Superman theory. So how many times in our environments do we have that one person that looks like Clark Kent every day of their life, right? And the emergency happens and boom, they tear open their shirt and they're like, I'm Superman. Well, think about employee turnover. Lately, it's absolutely skyrocketed by choice or not. All of a sudden, you're just used to a Superman being there. But remember, you don't know who Superman is. You've trusted he's in your trenches. What if Superman went to work somewhere else? Now you're in the middle of the emergency. Superman's not there somewhere like you're in trouble. So you're right. It's documented. So now it's the people in the process side. It's not just I bought software. It's written down. It's executed, it's tested, and it's ever organic, growing as we do new things. Every new system, every upgrade is, what do I do for DR? Is it backed up? Is it protected? The end. You know, that reminds me of the story. I was working at Lucent Technologies, and we had a guy on our team that was a cyclist, and he got hit by a truck and killed. It's very, very sad. He was a critical person on our team and he had a lot of the process on building our software in his head. Oh, yeah. Terrible. We coined a term called the trucking factor. I don't know if you've ever heard of it. Yes, yeah. Trucking factor of one is dangerous, right? One person has the knowledge or one and and that's very true in disaster recovery if you got one person that knows how to do the disaster recovery that's bad you need another person or you need at least documented so we always said one and a half minimum half is documentation but two and a half is a whole lot better than uh you know one and a half having having that there it's a sad situation but these things do happen in life, right? Especially if there's a true physical disaster, you want someone, anyone to be able to help you get back to your business, back to where it needs to be. Well, and I think that's where kind of I found my niche in the world was looking at fractional backup and disaster recovery. So rather than hiring the full-time person, you're going fractional, You're paying a fraction of the cost. Somebody's spending some time on your site, if you will, virtually. And this person's working and they're specialized target. And you may not need them for 40 hours. You might need them 10 hours a month. However, now multiply that and to say, well, when you hire a firm, maybe you're getting three, four or five fractional people. So you don't have that trucking factor. So not only did you fill the seat for less, but now you have the elimination of trucking as long as they're transparent and documenting. because don't even trust your DR vendor. If you hired me tomorrow, I'd be like, the worst thing you can do is trust me. You should respect me, but zero trust. I'm not insulted. You're right. Yeah, zero trust, but what was the word you used? Zero trust, but... But respect. Respect. Right. There's a new shirt. I'm going to do a new t-shirt. You can get it on my t-shirt shop, right? Zero trust, but respect. I kind of like that. now and and i see where you're coming um ultimately it's my business i'm responsible right so if i don't want to spend the money then i shouldn't be mad that i lost everything because i decided not to spend money to do it right well and look at your contracts too from an executive position when legal's going through and redlining i guarantee they're looking at a lot of sass vendors with boilerplate contracts. First of all, my corporate attorney would tell you, you can negotiate several of those things people don't understand, but come back to the value of loss. If you're paying a vendor a couple thousand dollars a month, and let's say you're a pharmaceutical company, and I had one in this situation, $6 billion worth of intellectual property, and they're spending $1,000 a month. And right away, they're like, well, what if we lose our data? You're our last great hope. Well, for $1,000 a month, nobody's going to insure your data for $6 billion. But even if they wrote you a $6 billion check, you're basically just cashing out at this point. You're paying everybody else, paying everybody, walking away and trying to recreate. So you have to look at your limit of liability. And you mentioned about your incident. I can tell a different persona from a person who has never been in an incident and how much they're willing to spend and what they're willing to go through versus the person who's gone through it, which like you said, unlimited budget. I don't care how long it takes. Get us back alive. Yeah. The incident is worse than the preparation. Yeah, absolutely. So how do you determine how much to spend? Because, I mean, like you said, I could spend an unlimited amount. As an executive you just weigh that risk I think probably But where do I start That the hardest part i mean it your revenue you have to figure out how much you losing per minute hour whatever okay they're very large manufacturer uh they're all over the world and one of the things that i've noticed about them is they have surprisingly low return to operation because their factories are segmented off if a factory is down it's not really in my area if a robotic arm goes down or something but they have their own backups but for those systems but let's just say that data isn't flowing into the factory well they're pre-loading seven days worth of manufacturing so they essentially could have seven days worth of time so do they need those cutting edge systems no do they need the secondary redundant crazy levels no they just need to say listen if this goes down can somebody get me this piece of equipment within 24, 48 hours? Can somebody rack it up? Can somebody recover back two days, three days? We're okay. If I'm running a dot-com processing transactions per hour, you know, if I was like PayPal or Stripe, think of how much they do in a second. They need to have it. And so it comes down to that business impact analysis. What does it cost us when we're down. What now? When we build Utopia, how much does that cost? Can we roadmap to Utopia and do it in stair-stepping over a couple of years and realizing that, listen, these are key critical systems. Like you said, email was everything. Well, that's user shock, right? It really wasn't generating revenue per se. It's inconvenient, of course, but on the flip side, you know, what was the revenue driver for that business? So it's coming down and understanding your business. Then it's coming down to what systems generate those revenues and designing the architecture of first up, second up, third up, full role, if you will. And that's where it's then going to come into your planning of, I mean, you have to think about where are people going to work? We all assume at home, but do they have access to those systems? That's the planning side of it is what will we do sort of thing. Well, I think it's interesting because I'm wondering if COVID really helped people understand business continuity better because holy cow, all of a sudden worldwide, everyone had to start working from home. Like overnight. Yeah. They sure did. And our phones ran off the hook overnight. customers that we couldn't beg them to test anything like you've been a customer for five years can we please spin up a sandbox to test these things so you can see wouldn't return our calls we're suddenly beating down our doors yeah we need it now we need it now and it was like yeah you do i wish you would have planned so it's not an emergency on my part but you're right that's scared a lot of people however they're starting to get lax they're getting lax again Yes. Yep. Yeah, I'm seeing that. Yeah, I think you're right. We've seen some of that happen. So this constant continuous DR investment is something that you should write into your business plan, it sounds like, and say, hey, it's almost like buying insurance, almost. Correct. Got it. It's buying insurance, but you're the one executing the insurance policy. When you have an insurance and you have to beg your vendor, please pay me at least here. You've kind of it's like self-insuring is what I would say. You've got this chunk of money sitting aside and you need to go to the doctor. You just draw off of it. And every month you're putting it back. You know, it's there. That's what they are. Is it self-insurance? Because if you run into a ransomware situation, this is very important. You know, first of all, it's very randomized. Nobody is targeting you. If they are, your your goose is cooked. They've been doing it every which way. But when we come into the randomize of a ransom and they want to be paid, the fact is we're finding out several things. Number one, even when people pay now, they're not giving out the codes. They're like, hey, oh, you gave me a million dollars. That's awesome. Now I want to. So you're lucky if you even get your data back. Second, you're now targeted because you're an own payer. So everybody's like, hey, go after Tom. He just paid. Awesomeness. So you have to be able to build your insurance with this. And one way or the other, you're going to pay. and this is a risk minimization. It's not a fun and sexy topic. When you put out a new web interface, everybody loves it. When I do my job, nobody knows I even work. Well, if you do it right, yeah, you're invisible. Exactly. Right. Interesting. So how do I prevent... Ransomware is a scary thing, right? Because they've infected my data, they've quarantined it, they've encrypted it and all that. how do I know how far I need to go back to bring back my data so they're not just right back in again? Well, that's the challenge. So modern systems, and I'm a big Veeam proponent just to let everybody know there's some key players, but they've really done a solid job for the longest time. But what I would say is a lot of these technologies, including Veeam, they have the ability that the moment you're taking a backup, it's mounting it and scanning it and it's looking So they're checking. So that's the check for malware as you're storing your backups. Correct. That's a big factor to it. A lot of times what I've seen is more of you've been surveilled for a long period of time and somebody now decides to enter in through a back door and now they begin the encryption. So sometimes it's time delayed fuse where it's automatic. A lot of times it's, oh, I've gathered enough information. Let me enter. Once I enter, let me impersonate. Oh, now I'm Darren. I have Darren's God rights on the network. Now, what's the first thing I'm going to hit? Those backups. Because they know if you have insurance, they need to deplete your account. So that's where we get into the specialty of backups and saying, how do we protect our backups even from ourselves? That's kind of that tier two. Got it. All right. This has been bringing back memories, a little bit of PTSD from being CIO. Right. Because everything, the whole weight, and people don't understand this about CIOs, the whole weight of the organization is on your shoulders when it comes to information management. And it can be difficult, especially nowadays. So thank you, Tom, for coming on the show, going through this. If people want to learn more about what you do and how that all happens, where do they go to find out more? So we have a website, differentdev.com. So it's like differentdevelopment.com. You can find me on LinkedIn, Thomas J. May. I'm out there under Different Dev. clearly we'll have some backlinks between us easy email tom.may like the month at different dev.com so i'd love to speak to you i don't charge for every minute that we're out there it's prospecting and learning what you need i've learned sometimes in business darren you know you make a connection you give someone a penny's worth of advice and three years later they've earned your trust and they come back so no if you're worried about it let's just have a conversation and see. It doesn't cost you anything other than time. Well, hey, Tom, thanks for coming on the show. This has been wonderful. I appreciate being here today. Thank you. It was a great time. Thanks for listening to Embracing Digital Transformation. If you enjoyed today's conversation, give us five stars on your favorite podcasting app or on YouTube. It really helps others discover the show. If you want to go deeper, join our exclusive community at patreon.com slash embracing digital, where we share bonus content and you can always connect with other change makers like yourself. You can always find more resources at embracingdigital.org. Until next time, keep embracing the digital transformation.
