Powering the AI Inference Wave with EPRI's Ben Sooter - Ep. 292

Foreign.

0:00

Welcome to the Nvidia AI Podcast.

0:10

I'm Noah Kravitz.

0:12

Today we're talking microdata centers with Ben Suter, Director of R and D at epri, the Electric Power Research Institute. The relationship between AI data centers and energy grids is an increasingly important one,

0:14

to say the least.

0:26

In a moment, we'll talk about how microdata centers can help strengthen that relationship. But first, a quick note about GTC San Jose. Join us at the world's premier AI conference. GTC San Jose is online and in person March 16th through the 19th. From physical AI and AI factories to agentic AI and inference, GTC 2026 will showcase the breakthroughs shaping every industry. Learn more and register@Nvidia.com GTC Ben Suter, welcome. Thank you so much for taking the time to join the Nvidia AI Podcast. Really glad to have you here.

0:27

Yeah, great to be here, Noah. Super excited.

1:03

So, Ben, to kind of set the table before we dive in, for listeners who don't know epri, can you briefly explain, well, first, who you are and

1:06

what you do and as part of that, what EPRI is and what EPRI does?

1:14

Yeah, absolutely. So EPRI is a sort of a unique organization. We're a 501C3, not for profit. It's an independent institute focusing on R and D, collaborates with more than 400 companies across more than 40 countries and really drives innovation to ensure sort of the public has reliable and affordable energy. So really awesome mission statement and been a really exciting place to work.

1:17

You've been at EPRI for a couple of decades now.

1:43

Yeah, I've been here a while. I just crossed over the 20 year mark, which I feel like is ancient times in the way the corporate world works now.

1:45

Right.

1:55

Well, congratulations.

1:55

And you kind of, this is exactly why I asked because thinking about data centers and AI, but hearing you talk about things like nuclear and thinking like,

1:56

man, you've like must have seen some

2:04

things and worked on some projects and thinking back, you know, over 20 years and how technology and energy reliance and consumption must have evolved. I don't know if this is a

2:06

fair question to ask, but can you

2:16

kind of place our current moment in context to, you know, sort of what you've seen with how the world uses energy and stuff you've worked on over the years?

2:17

Yeah, yeah, that's a great question, a great way to frame it because. And it gets to why I probably have stayed here 20 years, which is that there's just so much change and a lot of different Exciting things that have evolved across the sector and the industry that have sort of landed us here today. So lots of stuff going on. It's been interesting as you come in. And I'm sitting here in Knoxville, Tennessee, and behind me we actually have a big laboratory that makes up the back half of the building. But pairing with that over the years here at epri, I've seen all kinds of technologies come through, whether it's, you know, solar energy, battery storage, electric vehicles, all kinds of things. And what's it interesting is you see a lot of it several years, like a lot of times in advance of when it's cool and it's blown up and it's everywhere.

2:28

Right, right.

3:20

And so it's, it's been interesting to just see all of these technologies come in and evolve and the challenges that come with them. You know, whether it's, you know, how do we, how do we handle the loads of electric vehicles or how do we position the distribution system to handle all of the solar capacity.

3:22

Right.

3:41

All these different issues and then meeting those challenges. And so that's, it's been an exciting place. And I've had, yeah, I've gotten to have several lifetimes here because I've been here for 20 years and so working through different areas and now kind of in this AI space, which is obviously just accelerated everything about 10x.

3:42

Right, right. So let's get into that then. Data center. AI.

4:03

You know, when I say people, when I think of, you know, AI and energy consumption data centers kind of pop to mind sort of immediately. There's more to that obviously. But can you kind of set the stage a little bit for, you know, kind of explain what a data center is in this context and then maybe that can get into what this idea of a micro data center is and how it difference, you know, how those differ from the kinds of things that, you know, people like me usually think of when I hear data center.

4:06

Yeah, so, so good question. And you know, I think there are, there are several flavors of data center at this point. And I think kind of the, the two I'm going to sort of hone in on today, one is the, the data centers that have been really in the news a lot lately, these multi gigawatt behemoths that are being built with the objective of providing platforms to train these really exciting AI models. And so, you know, there's been an enormous push to build those, those types of capacity. Obviously there's been a big crunch for power in order to meet that, that demand. And so lots of exciting research in that area. But all of that's really been directed at making the models that are going to potentially do exciting things for us in the future.

4:35

Right, but the training of the models.

5:25

Exactly, the training of the models. But you know, and you mentioned this, you know, in the plug for GTC at the beginning, inference, I think, you know, people don't realize that while we were so focused on training the models, like there's this huge wave that's coming of once we actually get all these models and we move beyond just chatting with ChatGPT and we're doing the real time translation in our AirPods and we're doing the smart glasses and we're doing all the full self driving and all these different applications that all that, all those applications all falling into inference, sort of using the models is going to sort of accelerate this second compute wave that comes along with all this in order to have the compute capacity to actually do all this stuff. And there's actually a, there's an interesting statistic out there that if you look at like the lifetime of a model, so if you look at a, you know, a GPT 5.1 or whatever, only about 20% of its like compute capacity and thus its power consumption is in the training side. 80% of it is in the inference side.

5:27

80%. Okay, yeah.

6:33

And so the vast majority is actually in the inference side. So if you think about how much capacity we're building for training, we're going to need, you know, a couple of times that to meet the demand for, for all the inference.

6:35

And so people start using these things.

6:45

Exactly.

6:47

Or close to full max. Yeah, yeah, yeah.

6:48

And so, so that's going to create

6:50

another challenge thinking about energy consumption.

6:51

Is the distribution of energy consumption during inference as opposed to training, is that just massively different and much more spread out? What does that look like from the perspective of, you know, energy load and consumption and figuring out how to try to balance things?

6:55

There's a lot of great questions in there and a lot of I try

7:13

to throw 12, 13 of them at you at once, you know.

7:16

Yeah. And a lot of them are things that we're looking at as part of this microdata center project.

7:19

Okay.

7:23

So when, you know, when the world, when we got into these gigawatt scale training data loads, nobody really realized or thought about the fact that these, the way the compute and things would happen is, is the training loads would slam hundreds of megawatts of demand nearly instantly within milliseconds. And they can also fall off once that job is Done. And so huge swings of power. And that created some consternation as you had to solve sort of the technical challenge of meeting those demand peaks and spikes and things. You compare that to inference. And when we got into this, we started down this journey about midway through last year and you know, I was initially imagining this and I'm thinking about, okay, if inference is what I'm using, you know, one of these awesome models, I'm using ChatGPT, I'm using Groq, I'm using Gemini. And so it's being, the compute tasks are being generated by me. So that's going to give it what we call more load diversity. It's going to kind of smooth it out because it's being randomly generated. Sort of my initial hypothesis. Really interesting discussion last week with someone as we started bringing up just the whole agents and agentic AI that has taken over in just the last few weeks. You know, bring open claw at the house, you know, to take over my world. And, and I started realizing like, oh man, like that's doing all this work at night now.

7:24

Right.

8:56

So, so when I originally thought this load was going to sort of look like a normal load curve for just people waking up during the day, putting on lights and you know, air conditioners and stuff, now all of a sudden I'm like, well, that completely changes the paradigm because now it's running at night while I'm sleeping and. And is it going to do more? And, and so I. You, to answer your question, this was a really long winded way to answer it.

8:56

No, this is.

9:18

I want to go deeper and ask

9:18

you what you're doing with openclaw, but maybe that's another podcast. So.

9:19

Yeah, that may be another podcast because that's, yeah, that's trying, trying to streamline the, how you actually survive in the 10x corporate environment.

9:22

Right.

9:32

But all that to say that that paradigm sort of evolving now and I'm having to change my hypothesis. And so when we get, you know, when we actually start monitoring these data centers and things and actually building them out and realizing and measuring them, it's going to be really interesting to see what they look like. And I have a feeling you're going to see lots of different loads because it's something that's very consumer centric. May look different than. Yeah, there were some, some great stories last week of like some big financial institutions that were very AI forward and have, have invested a lot in models and don't have enough compute for their internal models.

9:33

Yeah.

10:13

Which is another. Is a whole Nother, you know, it fits very well into what we're looking at here, but it's a completely different probably, you know, shape and.

10:14

Yeah, yeah.

10:24

Well, let's dive into what we can

10:24

kind of grasp at the moment or, you know, is concrete, I should say, at the moment.

10:26

And this idea of micro data centers,

10:30

can you, you kind of alluded to it in talking a moment ago, but

10:32

can you talk a little bit more

10:35

about what they are and why now

10:37

and what are some of the problems

10:39

and these may be some of the examples you're mentioning. Are you trying to solve for the power grid as well as for AI users with this idea of microdata centers?

10:41

Yeah, so great question. So the real thing that we're looking at here, and I mentioned everybody's focused on these big giant training data centers. Now we're thinking about how do we create these data centers for inference. And when you actually look at those data centers for inference and one of the things you start to realize is that having the huge mega data centers that are centrally located don't necessarily make sense for the inference data centers because they are more consumer centric and user centric. Positioning them geographically around where, you know, the people are tends to make more sense because they can be more latency sensitive, et cetera. So you don't necessarily want to have them just in one place in the middle of nowhere. Better to have it broken apart.

10:50

I don't know, I may be way

11:43

off here, but it reminds me of when streaming media centers, you know, started popping up kind of in the whatever period of the aughts, I guess. Right. The first.com wave of when multimedia become a thing and yeah, that kind of proximity because it affects performance, as you said. So.

11:44

Yeah, yeah, exactly. When, you know, the early years of Netflix, where it started off very centrally and then they start, they realized, hey, if we put a mirror onto the local networks, it becomes a lot easier to distribute. So yeah, yeah, it's another thing. And incidentally, the, the biggest user of geographically dispersed servers. Game servers. Yeah, right, right, right through this like journey, like learn that little tidbit.

12:01

Can you walk through a little bit what happens in a microdata center in

12:30

terms of sort of, you know, how

12:34

do you design and build for an

12:37

inference load as opposed to a training load? And what does that mean in terms of both the energy usage, but then also like the ripple effect of not housing everything in these central giant megawatt data centers that as you said, at least for training, you know, they act differently than other big loads on the grid they come up super quick and big, you know and I imagine all kinds of other problems that are beyond my knowledge set. But just can you talk a little bit about how they sort of work on that level?

12:38

Yeah, so, so a couple of things that are kind of in that, that onion to unwrap. So you know the first one sort of on the really underlying underneath construction, it's somewhat similar in the fact that it's still very sort of GPU or TPU based compute need in order to actually run these models. We're seeing, I think more chips like Nvidia has more chips more designed for inference and training now. Um, so there seems to be a little bit of diversification where it was now just sort of one, one chip initially. Right. And so we're, we're, we're seeing that. So there, there is some variability I think maybe in the underlying chips. But, but traditionally it's been sort of the same chip that for training and inference. And so from that perspective it looks similar, it's just smaller because I, I don't, I don't need as much. But you know, I, it's as sort of the result of, you know, I don't need as much. It's sort of like how much do I need? And that's, that's been one of the things that we've been looking at and one of the interesting things that we've been working with technology partners like Nvidia to really help us understand, you know, what the compute needs of the actually technology companies that are buying these data centers, using these data centers, looking in that, you know, is, is 3 megawatts enough? Is 5 megawatts enough? Do we need 20 megawatts? And there seems to be, we seem to be coalescing somewhere around this idea of 20 megawatts, but that's actually sort of, I hadn't gotten into some of the electrical aspects of all of this. But as we're looking at where to place these microdata centers, 20 megawatts can be not insignificant ask of just dropping a load somewhere onto the grid. And so there's not a lot of opportunities to drop something of that size. Okay. And when EPRI was looking at, okay, you know, our partners are telling us about this, this coming compute wave and we want to, we want to do what we can to help our utility members be proactive and get ahead of it. Where can we look at opportunities to find power for this type of data set?

13:08

Right.

15:26

One of the things we started looking at was, well, there's substations all over the United States and indeed all over the world. And there's a fair number of them that are actually underutilized. So they've got excess capacity available inside them. And so we started thinking like, well, is there an opportunity there to partner with those substations that have that excess capacity and do something and, you know, put these inference data centers near it and, you know, maybe directly adjacent is maybe ideal, but close by and make sure, you know, we've got everything that is needed in terms of fiber access and.

15:27

Right.

16:05

All the infrastructure, all the underlying infrastructure. And so look at all those things and say, does that work? And we thought that was a good idea, but the answer is you're probably going to find 3 to 5 megawatts, maybe up to 10 megawatts of available capacity in a single substation. And so then we started thinking about, well, how's that going to work?

16:05

Ben, just to interrupt you real quick, sorry.

16:30

Because I keep having a picture in my head of. And this is my own ignorance about

16:31

our electrical grid of how big one

16:35

of these existing substations is and where it might be. Is this kind of like suburban as opposed to Metropolis?

16:38

Is that so? It could be both. Okay, so there's a couple caveats in there. So you're right in thinking that your suburban substation may be more likely to have some of that excess capacity. Okay, that said, we have found that there's interest at the Metropolis level, too, in capacity incapacity, because there is. There is need. You know, there's people there, so they want to get the compute close to it. And actually, if you see, you know, some of the Metropolis environments, there's a lot of real estate that's available right now.

16:46

Right, okay.

17:18

Which, which equates to load that's not there. So there's. There's opportunity to put load. So that was another hypothesis going in. Yeah, there wasn't going to be interest, but actually, it looks like there may be interest and opportunity at that level as well. And so as you're looking at these data centers and you start to say, well, does three megawatts make sense? And does it make sense for the person that wants to buy it? What we realized was maybe there's an opportunity. And this is the distributed part. We initially kind of called this project distributed inference, truthfully. And while distributed inference seemed to be very technically accurate, it did a really poor job of giving anybody a visual image of, like, what it was we were talking about. And so what we realized was, if we go to an opportunity. If we go to a regional area, we go to a city and we say, hey, is there other five data centers that meet this criteria? And then, you know, each data center maybe has 5 megawatts of capacity. Now we've got 5 data centers at 5 megawatts, and now we've got 25 megawatts of capacity. And so actually looking at it as, you know, instead of a single project that's 5 megawatts, looking at it as a 25 megawatt project that just happens to be distributed across five sites. And so that helps meet the needs of like, what the utility grid has available and sort of meet the economics of what the data center companies need in order to actually make it, you know, realistic and viable for them.

17:19

Right, right. How does this approach affect the way

18:51

the grid functions for just, you know, people in general, the. The city, the region in general?

18:54

So, so great question. And, and we've really sort of seen this as. As a win, a general win for everyone. Because the answer is if the. The existing substations are already kind of sunk cost, we've, we've invested that capital. Yeah, we've made the investment, we've built it. And so if we can get, you know, extra capacity, if we can get extra usage out of existing assets, then that's sort of a win for everyone. You know, if, you know, a societal cost, if we're not having to put new steel in the ground, then that. Absolutely, yeah, that's helping, you know, keep rates lower and things like that. So we really see this as a positive in terms of being able to leverage existing infrastructure. Speed to power, I think, is also a big part of this where there's a huge scramble for this capability and everything. And so it also means that you no longer have to deal with interconnection queues because you're off the transmission grid and all the things that go along with that. So definitely speeds up the ability to get to a finished product that's online and serving customers much faster as well.

19:00

That's great.

20:13

Are there clean energy implications?

20:14

It's interesting you say that. So definitely there's opportunities to layer all kinds of things on this. So there's opportunities to layer this with dermatology and solar, wind things. And I think there's also a lot of opportunities for energy storage. One of the things we've been looking at, getting sort of into the technical weeds, we've been looking at flexibility and how what you find is that you'll have a substation and it's got excess Capacity, but it's actually got quite a bit more capacity. Except for July 21st when you have the hottest day of the year. Right. I'm making July 21 up. That's not the hottest day of the year. Somebody fact checked me.

20:18

I was like, wait, what AI breakthrough happened on a July 21 just been made, right?

21:04

Super hot day.

21:11

Yeah, yeah.

21:11

So if you, if you're, if you have, if you can engineer it so that you can have flexibility to reduce your load, you reduce your demand during those peaks, you actually have a lot more, you know, envelope that you could potentially use. And so pairing it with energy storage, backup generators, just working with the technology partners. One of the other nice things about if you have sort of a distributed network of these loads is if there is, you know, possibly like a peak demand issue, I can run down my compute on way to center and route the, route the calls someplace else.

21:12

Right.

21:48

And move smooth things out that way. So there's lots of possibilities. And so that's another thing that sort of makes this exciting and a really neat way that a tool that the utilities could use as well.

21:48

Yeah, yeah, no, that's very cool. Continuing sort of along the lines of the applications of all of this, but kind of from the other side of it. And again you talked about this in reference to, you know, building the data centers, these smaller data centers close to where the users are, the consumers are and that performance aspect of it.

22:01

But are there other examples of real

22:20

time applications that as this infrastructure rolls out, you think will be enabled or maybe just kind of accelerated, these applications that could directly benefit people?

22:23

I think there's all kinds of things and I am certainly not going to claim to have a view into all of those options. You know, I mentioned some, you know, like the translation and self driving and things. But you know, I think especially as agents develop as we get, you know, smart glasses that can analyze just here at epri, you know, other exciting things, you know, we're looking at and these are going to have applications for everybody. But you know, can you use smart glasses to analyze your poles and transformers and, and things in a substation and make, you know, your line workers smarter, more efficient and safer all at the same time. And so, you know, there's all these applications that everyone's looking at. Can we, can we, you know, again grid focused? But can we, can we make the control center of the future smarter? Right. And get smarter about restoration times and all these different things on and on. I think there's just internally at epriat, there's a few hundred app use cases and things we've identified and that's very grid centric. So obviously the audience is probably not all utility workers and things, but I can only imagine that if the electric industry has identified several hundred use cases, then around the world there's gotta be just tens of thousands.

22:37

We wouldn't be here having this talk on tape, so to speak, if there weren't. Right.

24:03

Kind of.

24:08

I was just thinking about this as I was listening to you and you spoke to it with examples of like smart glasses with people out, workers in the field you know, analyzing things.

24:10

But are there ways that you've seen,

24:19

you know, and whether they're. You're using them now or maybe things that you kind of see coming that you're excited about, ways that the energy industry has been using AI to and I don't know if it's like to design better battery storage or to explore, you know, new forms of energy or to, you know, maybe something seemingly more mundane but still really important like reorganizing the way that, you know, companies approach different industries.

24:22

I don't know what. But are there big examples that kind

24:54

of jump out in, you know, your own work or what you've seen of how AI is transforming the industry from the inside?

24:56

Yeah, I mean I think it's transforming it in all kinds of different ways and it's one of those things that I think has been, it's been really interesting because things do seem to. There's lots of memes about how fast things are going and I already made some comments about 10xing and things. But it's all sort of the proof is in the pudding. And have we seen where's that scaled demo? I think there's a lot of proof of concepts that we're seeing pop up around and really the thing everybody is waiting for is that scaled demo of where there's this application and it's measurable and we've scaled it out to the entire enterprise. So there's definitely a lot of work to do. But I think there's lots of applications as well. Yeah, it's trying to go through my head there's just so many different things but, but you know, every. Because everything from understanding, you know, in the utility industry, there's a lot of historical records and things and several of a lot of them predate sort of the digital era. And so current models and things can make just ingesting all of that and structuring it into to useful structured data sets that you can then use to create New models and create analysis and digital twins and all these things. So that I think that's. There's some of the places the existing work is already really useful. Obviously all the things we do every day just to accelerate ourselves with understanding emails and figuring out how to have that hard conversation with the problematic co worker. Making these up as well?

25:02

No, but it's related. It's that. Well, it's that interesting. Sort of.

26:50

There's two layers. Well, there's many layers. The five layer cake is the. The iconic layer at the moment. But there's kind of two layers. What I'm thinking about, there's the layer of like the kinds of work that I don't want to call it knowledge work, but that kind of working with information you just described that is part and parcel of many roles in many industries. Right. And then there's kind of the NAI is helping, you know, helps me day to day in ways you were just describing or you know, kind of making up. But I get you.

26:55

And then there's that layer on top

27:23

which is specific to the kind of work and the industry that you're doing. And the more people like you I get to have these conversations with, just the more in my mind I see like, you know, it's both. Right. And one informs the other. Being able to go back and ingest all that old data, you know, we've had as a cardiologist or a radiologist on a while ago talking about how much hidden information there is in old analog film scans.

27:25

Oh yeah.

27:50

That you know, AI image analysis is able to extract now. And it's useful. Right. And that kind of stuff is. Yeah.

27:51

Did you see the guy with the microfish like repository?

27:57

It rings a bell, but I don't know that I did.

28:01

This is a few months ago now, which makes it ancient news. But yeah, there was somebody that had access to this huge repository of microfish. And I'm old enough. Those of us that are old enough on here, we'll remember looking at it under the little magnifying contraption.

28:03

The machine in the library.

28:19

Yeah. To see the News article from 1942. But he had access to tons of this stuff and started using the models to ingest it all and just created a monster data set and so cool.

28:20

That's amazing.

28:33

I love stories like that.

28:33

All right, Ben, as we get to

28:35

kind of wrapping up here, so I

28:36

can let you go. This is not to put you on

28:37

the spot because as you mentioned, these kinds of things are impossible to. It's always impossible to predict the future, but when things are moving as quickly as they are, it's harder.

28:40

Right. But if we look ahead to the

28:49

next year or so loose timeframe, what

28:51

does success look like with microdata centers

28:54

and even more broadly? I guess that's what I'm thinking about, putting you on the spot both for the grid and for everyday users of AI powered services.

28:58

So great question. So I'll start with the microdata center part since we're talking about it. And you know, I think, you know, hopefully in a year or two years we've got a pile of, of of these, you know, micro inference data centers built out and we're monitoring and measuring them and that's helping educate us on what we need to know so that we can continue to build them out for all the wonderful things that the industry is going to create. So I think, you know, from the microdata center standpoint, you know, that, that I think is, is what I hope, what success looks like.

29:06

Yeah.

29:40

And then, you know, I think just in general, you know, I have no idea that everything is so exciting. It's, you know, you mentioned GTC at the beginning. I learned something new, you know, from, from, from those types of conferences and stuff. Every year there's new things that come out, completely change things. I mentioned agents, you know, which. Yeah, which are just like weeks old, maybe a couple of months old that we've really sort of delved into that it's changing the landscape again. So, you know, I don't know what it's going to look like, but I'm hopeful and you know, it's going to be exciting and there's going to be compute needs. You mentioned, you know, at the very beginning, sort of the importance of power and stuff. You know, I think, you know, there's, there's still going to be challenges to solve to make sure that we can provide all these awesome things to everybody and really move society forward and everything. So exciting times.

29:40

Excellent. Yeah, well, I, I'm with you. I'm rooting for you and I'm excited to see how it all unfolds.

30:32

Ben, for folks who would like to learn more about the work you're doing, about the work EPRI is doing, where are, where's a good place for them to go? Online? Website, social media accounts.

30:38

Where should they start?

30:49

Yeah, absolutely. So website. So you can go to EPRI.com, ePRI.com is our official website. So lots of great information there. Also very active on LinkedIn. There's lots of, if you're interested into the latest news about exciting AI and data center updates sort of in their adjacentness to the electric sector. Lots of good stuff going over there on LinkedIn, so those are probably the two places to find us.

30:50

Perfect. Ben Suter, thank you again for joining the AI podcast and best of luck

31:19

with everything you and everyone at EPRI is doing.

31:24

Appreciate it. Great to be be here. Great talking with you. Sa.

31:27