Mechanical Engineering at Oxide [chapter images]

That's Elliot. Elliot is here with me in the letterbox. Hello. Hello, hello. I just thought of Brian's gotten really good at doing impressions. I, you know, I have been polishing. Are you saying, are you talking about my Elliot impersonation in particular? Yeah. Yeah. It's very impressive. I would have never known. I do think, okay. I know it's rude to speak of Elliot in the third person here, but I actually feel that like Elliot has a voice that I could impersonate. It's funny. You should mention this. I'm like, this is something that's been at the – Adam, you and I had a CS prof in college. And I'm like, this guy is – I think I can impersonate him. I think that there is – there are some – I'm going to say, Elliot, delightful idiosyncrasies just to be – unfortunately, we're not on video, so you can't – I'm trying to convey Elliot's facial. expression right now, which is definitely a... You can just forward this directly to HR. It's not a big deal. Yeah, I think it's taking a wait-and-see approach. I would like to say that I broke some new ground, Adam. I felt when I started to impersonate him, it was not you, Elliot. I haven't yet started to impersonate you, but Elliot looks so uncomfortable. I kind of... I opened the sluice. It was a floodgate. 100%. And I think in the same way that George Bush first, Dana Carvey's impersonation of him is then what people impersonated. What people impersonate is they – I think you're right. I think people impersonate Dana Carvey impersonating George Bush because he kind of exaggerated all these things. That's right. I think in the same way when folks are – It's possible in every man. Exactly. And so when folks are impersonating this professor, really, they owe a great homage to you. So, Elliot, this is what you have to look forward to when I really polish this lurking impersonation of you that now I've given. I'm so sorry. Brooks, how dare you open this Pandora's box? This is what we call it when a Pandora's box of discord. I mean, it's like the golden apple of discord meets Pandora's box. I think that's what was in Pandora's box. That is what Pandora's box is, basically. Yeah, yeah. I don't think it was like goodies. I don't know if you got that one wrong. It's not like a chocolate box. Pandora's box, from a Greek mythology perspective, Pandora's box contained the golden apple of Discord? No, no. It just had Discord in it. All of Discord. Yeah. Just in general. Oh, it just was Discord. Yeah. And not Discord. Personify. Although maybe that too. Yeah. Welcome to the podcast, mechanical engineers. Welcome, Elliot. It's like finally a box. We're talking about something I can enjoy. Thank God. It's great to have you all here. And Doug, Ben, great to have you as well. Elliot, provided that I haven't made you entirely self-conscious, do you want to kind of kick us off here about – actually, I'm going to back up a heartbeat because I – in terms of required listening. So maybe, Doug, you can give us some backstory of mechanical engineering at Oxide, and then we can use that as a segue to the modern era, to Brooks and to Ben and to Elliot here. Sure. We've had some great discussions with you on the pod about some of the terrific work that you've done before you came to Oxide, when you were drunk, and then after having come to Oxide. Yeah, absolutely. I can go through a terminology here. So, like you mentioned, I started out in a contract role here almost six years ago, working through some of the, you know, basic concepts and designs that led to our MVP, the minimum viable product here, which we're now looking to take, you know, to the next level. So we've got product in the field. We're looking to, you know, grow our mechanical engineering team and harden our product and grow our manufacturing volumes. So, yeah, like you mentioned, I've been on a couple of these in the past for cabling the backplane and the minibar system. And I guess we're kind of looking to expand on what we've done in the last year or two with our expanded team here. So all kinds of fun engineering challenges that kind of tend to change as we adapt to different situations for customer needs, scaling manufacturing. We change our hats pretty frequently here, and it's always exciting. So, yeah, that's kind of where we've been at. About two and a half years ago, I joined Oxide Proper as a full employee here, as did Brooks. And probably six months ago, we brought on Ben and Elliot. So we are growing, and it's exciting times for mechanical engineering at Oxide. It really is. And it's definitely worth going back and listening to some of those previous episodes. and Doug in particular, when you were, I believe it was in the cabling the backplane episode, where you were talking about all the things that we had done mechanically to prevent bent pins, to really make the, and I got to say, there were so many details that I just did not appreciate. And this is always the truth about like great engineering, right? It's like you ultimately, you got to sit with an engineer to really appreciate all of these kind of finely tuned aspects of the craft. But the net result is like, you think, oh, this just works. It's like, well, it just works because it's been very carefully designed. And, Doug, it was really great to hear in this kind of previous episode some of the thought that had gone in to really prevent, Adam, what you and I have called Adam Leventhal hardware engineer, which is to say, which I think we used to say involves a running start. Like that's one of the key arrows in Adam Levithal hardware engineers. Quiver is a running start. Absolutely. It's kind of the only one, but yeah, that's right. Right. We also often call this the max power way around here, which is like the wrong way, but faster for those of you who are the, your Simpsons Talmudic scholars. Presumably the max power way has got to be on someone's bingo card, but if not, it should be. um so uh maybe we can fast forward a bit Elliot um to because when you and Ben joined the team I one of the things I love about our mechanical engineering team is that folks are coming from pretty disparate backgrounds and you were coming from something that was like mechanically uh pretty interesting making ag machinery um and looking at oxide being like this is just like a box do you guys make like you guys are like silicon valley you're the silicon valley episode like you make the box it gives us but it's pandora's box it's great um do you want to talk about your kind of a bit of your background and and you're thinking about oxide this is my elliot impression i think i've honed it over the last few minutes very well played but yeah but in all seriousness hello everyone um first time on oxide and friends um yeah like brian was saying i came from directly from a company called Fido, rest in peace Fido. We were making agricultural machinery, agricultural robots that harvested aquatic plants. So a lot of moving parts, a lot of like interesting first principles design that we were doing. And it was like Brian was saying, more obviously interesting, but what I've really come to appreciate about oxide, and we'll talk about it more later, but is some of the similarities to some of the aerospace work I had done previously. Where when you're, I'm used, I'm very comfortable manufacturing in the like one to 10 quantity where it really needs to work the first time. And I've noticed a lot of similarities between that and manufacturing tens of thousands or soon even hundreds of thousands of parts where it doesn't need to work the first time, but it's really, really helpful if it works the first time because then you don't have a truckload of bad parts, for example, which is a costly mistake in time and money. Yeah. So a different kind of engineering challenge. A different kind of engineering challenge, but leads us back to some of the same relationships with prototyping and analysis. Interesting. Where the things that you get to care about are interesting and nuanced, like contact, for example. Okay. So say more. Or jump ahead to contact. But yeah, what do you mean? Yeah, what I mean by contact is, like you said earlier, we don't have a lot of moving parts in the rack. We have the fans and the sleds that slide in and out and doors that move. And that's about it. And all of those are not really moving in service. But when you think about the things that matter in the rack, it's all of the electronic pieces talking to each other and all the software talking to each other. And those things need to be correctly connected. And there was a problem that we probably won't go into in detail here, but it caused us to look at more of how our drives were mated in detail. And that problem ended up not being a mechanical problem per se, but caused us to look at exactly how things were touching under a microscope, basically. And that led us to some of the work that I guess we could show now. on the shark fin analysis. So you describe what shark fin is a little bit. Let's use this as kind of an, because we should say that symptomatically this kind of approached, we came to this because we were having PCIe errors. We were seeing more PCIe errors than we would expect on certain lanes. Yeah. And so the SSDs are connected to what we call shark fins. They're a separate board connected to our motherboard in the Cosmos LED. And exactly how connected those are and exactly how, I'll say exactly how connected they are, both in what we call the Y direction and the Z direction actually matters quite a lot. With typical SOLIDWORKS coordinates, Y is up and Z is forward and back in this case. And what we were looking at is like, okay, how far into these drives need to be and how far out can they be before they actually lose their connection or start to cause PCIe errors? And I think what Doug put on the screen here is a look at our original shark fin tolerance analysis. The uninitiated tolerance analysis is basically the mechanical engineer's fancy addition and subtraction with statistics, where you can actually look at the manufacturing tolerances and tracing them through how all the things are supposed to touch in the mechanical design. and basically adding up the distributions of those things and determining, okay, in our worst case or somewhat near to our worst cases, how connected are these things? And on the other side, how connected are these things? So how smooshed in or how smooshed out are these things? And what we noticed is that we were actually a little bit marginal on some of the stackups in that we were going through a lot of different parts and they were touching in ways that we expected, but those distributions were a little bit wider than we thought they should be. And the typical thing in the mechanical engineering world is to be, okay, we need these distributions to be tighter. Let's tighten the tolerances. And as we're scaling manufacturing, we really, really do not want to tighten tolerances wherever we don't need to because that imposes additional restrictions on our suppliers that are already doing a great job. We don't want to make their jobs harder. And also we don't want to increase fallout wherever possible. In fact, quite the opposite. We want to be loosening up tolerances wherever we can for exactly that reason of increasing manufacturing yield, like preventing places where we've rejected a part because we had a tight tolerance on it that didn't need to be or was like something we hadn't really done a bunch of analysis on. So, you know, very much the opposite of fighting tolerances, as Elliot was saying. We want to be doing very much the opposite of that. Exactly. And we took that opportunity here while looking at the shark fin and the SSTs under a microscope to figure out, OK, the way these are touching in that tolerance analysis that Doug dropped into the chat, it's actually going through the shark fin bracket, which is a metal part that connects directly to the board itself, and then up through a PEM insert, up through a screw, up through another sheet metal part, through a bend, through another PEM insert, through a screw, through the Cosmo chassis, through a whole host of other parts before it actually gets to the SSD and to the other side of that chain where things are supposed to touch. And that is a lot of things. So what we noticed is we could actually flip that stack over. I don't know if someone can drop that into the chat, the new stack. But by having that shark fin bracket touch off against the bottom of the Cosmo chassis instead, that cuts out a couple of bends and a lot of different components from that tolerance stack up. And thus, by simplifying that tolerance stack up, we've actually shrunk the distribution of fit conditions that we could be experiencing. And that makes our thing both more reliable and we could even loosen the tolerances a little bit and still get at least as good a result. That is really interesting. And I feel like this is a very common theme where you all discover like we can actually really, we can simplify this. And by simplifying it, we often make it cheaper to manufacture. We make it more reliable to manufacture. It feels like it's got all of these kind of these other benefits that often come along with by kind of distilling the problem a little bit. That's right. And actually, if you look at the sharpened bracket, increasing the length of the tab on the bottom about three quarters of a millimeter, which is a very, very small change, but that actually totally changes what is touching nominally in the way that we are. And what that enables us to do is we have a bend and a PEM insert in each of those sheet metal pieces, and we can actually remove those features because they're no longer needed. Yeah, that is really interesting. I want to jump back to something Brian said a second ago about taking a problem, like, DarkPens, I think, are a perfect example of this, taking a problem and really dragging it back to first principles and seeing if we can do something weird, different, out of the box to solve it. That was actually one of the things that originally drew me into Oxide from, like, an engineering excitement perspective, was just the opportunity to use all parts of the product. because when I joined two and a half years ago or so, we didn't really have a lot of them in the field. It was still very new, very fresh, clean, shiny hardware. And this is all still true today. But the pitch was basically, hey, we have this thing. We've taken the computer back to first principles, and now we're making it again from scratch completely. And in so doing, we've fixed a bunch of problems. We've done a bunch of really fundamental improvements, like Brian's favorite hardware example, the fans, honestly, fantastic example. There's a lot of things like that. And that kind of continues to be the, what's the right word? Not like the MO, but like that kind of continues to be the central theme of a lot of the work we're doing is how can we just fundamentally make this better by looking at it as a blank slate? Like, is what we're doing correct? not just how do we take the thing we have today and like incrementally improve it and iterate it yeah i think that also i mean one of the things that has been really fascinating to me about watching the way you all work is the amount of time that you really spend understanding how they're manufactured i mean i i mean you have you have to because it is like it's the physicality of the thing but that to me has been really interesting that like you are i mean and i guess this shouldn't be a deep thought that like, you know, Adam, in addition to World War II being stressful, mechanical engineering is very important for manufacturing. You can present that to your list. But I just think that like over and over again, I, I mean, Brooks, like you had a very concrete example of eliminating a bunch of hems from the, from the rack. Do you want to expand a little bit on that example? Just because this was, this is one of these where it's like, And I feel like this often happens with you all where you describe some aspect of the rack or of the actual sled that I didn't appreciate before. And then you start seeing like, oh, my God, yeah, there are a lot. It's like, yeah, how do you assemble this? Like, yeah, this really is a real pain right now to assemble or what have you. Do you want to hit on that a little bit? And yeah, an explanation as Adam's saying, I know, I'm like, I'm tossing around a PEM like I can know what I'm talking about, which I emphatically don't. You do now. Exactly. Yeah, absolutely. So what we're referencing here is a couple of weeks ago, I did a demo about this change I'll talk about and pivoted very shortly into the demo, realizing that I had not provided any of the background information that the audience needed. So we'll start with that this time. So PEM, they make any kind of fastener you could possibly want that can be pressed into sheet metal or like blocks of metal, that sort of thing. Um, in order to threads, add a stud, add like a keyhole pattern, add a knob, any kind of three-dimensional, like physical fastener or like out of sheet metal plane piece you could want to add, PEM makes. Um, the company is, is Penn Engineering. There's a hundred different companies that make these. This is the good one that everyone in America uses. Uh. They don't really have a serious competitor. So we just call them PEMs. If you ever hear someone call something a PEM, they mean these. Exactly. It's a thingy. It's a doodad. It's a bit that is not coming out of the original, like, flat sheet of metal that is getting laser cut or stamped or whatever and then bent up into shape, which is how, you know, every piece of a car is made, every piece of a tractor, computer, you know, whatever. Almost everything in the world is made out of sheet metal because it's cheap, it's easy to work with, but it's really bad for putting threads into because it's very thin. So on the original version of the rack, you might hear some of us call something Rack Model Zero or RM Zero. What we mean by that is the oxide rack as it exists physically today. That is the first version of the rack that was designed by Doug and team. And the intention there was to be really reliable and really solid. And that worked fantastically. And now that we're trying to go through and make a ton more of these, PEMs take a lot of time to install. And they take a lot of time to deal with and make sure that they're right. And then when you go to paint apart or powder coat apart, you have to cover them over. You can't just, you know, run paint through it. It'll get paint all up in your threads, which is bad. And so right now the rack has some major sheet metal pieces to it. Obviously, there's like there's a base, there's a roof, there's four big vertical columns of sheet metal, each of which I don't remember the exact way, but they're very heavy. Like one person can't reasonably pick one of these things up and manipulate it. So right now the start to finish fabrication process on I just going to choose like the front left vertical I think we have a picture of this in one of the in the in the archive Someone can grab that So this front vertical of the rack is a seven and a half, eight foot tall, big piece of metal. And right now, what our sheet metal fabricators are doing is they're cutting this big piece out. They're moving it onto something called a press break, which is a giant set of angled steel bars that will bend it into shape, doing some of the bends, taking it off, bringing it across the factory floor to another piece of machinery that looks like a giant, that's like a small-ish press, lining up all the hardware, again, on this like 100-pound piece of metal being manhandled by two people, pressing in. two i think our most pems in a piece is like 40 ish uh it's a lot it's a lot of individual operations every single one of those has to be set in a in a tool aligned to the part smashed set in a tool aligned to the part smashed times 40 per piece times four per rack times however many racks we're making uh which is a rapidly growing number and when your constraint is and we've said this for a little while, like our problem is how fast can we make them? Like that becomes a real, real issue because this is someone, this is someone who's on a manufacturing line. This is a person who's doing this. This is two, this is two, two full-time employees per part. And there's, you know, many of these per part and it's operations that are happening many times. So any change we can make is scaled times hundreds per rack of, of time improvements. And then they take this part that now has 40 or so pieces of kit in it, back over to the press brakes and do more bends on them, et cetera, et cetera, et cetera. So you can understand how this gets very ungainly, very fast for trying to make these quickly. So there's another kind of thing called a punch, extrude, and tap feature. It accomplishes the same thing. It's a thread. You can put a screw into it. It's great. It's easy. and then, but instead of having to take the metal all the way off of the machine, across the factory, monkey around with it, get them all set up, and then bring it back across the factory every time, it's done in the same kind of machine. So instead of pressing in a third piece of hardware, it's actually punching a small hole in the metal, and then it's like two cones. I'm gesturing with my hands, which is not helpful for a podcast, but you know, hardware engineer, what are you going to do. It's like two cones that just get smashed together, much like if you've seen a video of a car door getting made. There's a big stamp and it goes bang and then there's a car door there. It's like that, but it's, you know, six millimeters across and not six feet across. So it's much, much less loud. And it forms this cone shape in the metal by just hitting it so hard that it flows like a liquid. And then you can tap that hole normally and then you just have a thread there. And so this operation can happen in seconds as opposed to minutes. And again, times, times hundreds per rack. So it's like a very, it's like a very, very simple change of it's still threads. Like if you're standing four feet away from the part, it looks exactly the same. The function is exactly the same. There's more like upfront pooling and setup costs. So we've kind of gone through. I was going to ask Brooks, it's like, yeah, why does, why wouldn't one ever use PEMNOTs? It is because when you're doing a new product, it would require you to do a bunch of tooling around it. Is that right? Depending. So some punch and extrude features are pretty standard. When you're making 10 of something, there's different ways in which you're fabricating those parts. It's much more manual, much less automated. You're laser cutting out the profiles, and then someone is hand aligning them in a machine, bending them. And the machine that you're using there can't as easily do these features. But at the volumes we're talking about now, we're on automated turret punch machines, which are basically, it's like a large table with bearings that these things run around on. And it can line up a hole with one of these like conical punches and just go bang, slide the part, bang, slide the part, bang, slide the part. Whereas the like, you can do that with an operator, but it's a much less efficient process. and it's not actually faster, the low volumes, if that makes sense. Yeah, interesting. You can also get a little more design freedom with the PEMs. You can get more threads, you can get different materials, you have a little bit more design freedom with them, whereas with the extrude and tap features, you're pretty much at the mercy of your material thickness. There's only so much physical volume of metal in the space you have to work with, So you can't put like an eight millimeter diameter screw in a two millimeter thickness piece of steel. It just doesn't work. Like it's impossible. So we haven't gotten rid of all of that. Adam Levithal, harder engineer, are you paying attention to shaking the one down? Because I know that's the first thing you want to try. Yeah, exactly. Yeah. It's just going to rip out, but you can, you try. I've done dumber things in production that at small volumes. Yeah. Yeah. Oh, we should pull up the picture with all of me. So this is just a concrete example of how we made it faster to manufacture. Then you also, like, these PEM nuts are not super, super cheap. I mean, they add up. I mean, you have a bunch of them. Yeah, they're like 50 cents. Each PEM is like 50 cents. And I think the charge for, it varies wildly manufacturer to manufacturer, what country you're in, you know, cycle of the moon, that sort of thing. But my rule of thumb in the back of my head is 50 cents a PEM, 50 cents to put it in. So dollar a thread. The extrude and tap features are free. Right. Yeah, interesting. And for us, we're really trying to optimize for time there more than anything else at the moment. But the time savings is really significant. Yeah. So the picture I just dropped in chat is on the left. everything in the rack except for the PEM, or both pictures are everything in the rack except for the PEMs, hidden. So you're just seeing the ghost image of the rack. Yeah, this is the PEM-only view that Elliot put together. This is how the PEMs see their universe. They only know one another. This is the universe that they are in. That's right, the constellation of PEMs. So you can see how much lighter the image on the right is. And those two big chunks in the middle are sidecar, which we haven't redesigned yet. So that will all go away too. But every single little thought that is gone is a dollar off the bomb and hopefully only a minute off of production time for a rack, but possibly more depending on the part. A minute per PEM. And obviously that's like a skilled operator that knows what they're doing. I assume this is not... Yeah. And I'm amortizing that time cost over, like taking it off the machine, carrying it across the factory, that sort of thing. Each PEM itself is probably on the order of low double-digit seconds. But again, it adds up. And it's operator time. Yeah. Does it also increase errors? That is to say, I can imagine, obviously, if you get one of these wrong, it can kind of ruin the whole part. is uh is there some compounding errors or or kind of failure rate or fallout rate as you have more of these these being or these being pems pardon me um not more so that they're they're incredibly reliable um all all things get more likely to fail as you make more of things we we talk about like quantity one problems, quantity 10 problems, quantity 50 problems, etc. I'd say PEMS, assuming your sheet metal house is good at them and isn't cutting corners, PEMS are a one in thousands problem, you would hope, or one in hundreds problem, and extrude and tap features are similar. From a quality perspective, I think it's probably not too much of a win. It's mostly a cost and operation time. perspective and just like hardware simplicity like this makes there's a lot of cascading things down from here that we're not even talking about of like documenting this is easier and like communicating this to suppliers is easier because we just put the thread in and we just say yeah this is the thread we want please give it to us and they say okay we know how to put that thread in this metal versus like specking out the exact pem which involves you know picking the thing the material, making sure that we've designed the right hole into the part, making sure that it's oriented correctly, making sure that we haven't used weird dissimilar metals that are going to corrode and fall out, that are going to bind correctly into the thickness of sheet metal we're using, none of which is individually hard. It's all on a very clear data sheet because Penn is a great company. But it's just another layer of engineering thought and validation that has to go into it beyond, yeah, give me an M3 thread, please. And they say, okay, here you go. Yeah, interesting. And so that's obviously a huge one. Someone was asking if we were concerned about the divergence between the rack as we kind of evolve to rack model one and so on. I think a lot of these things are things that are not necessarily like really, I mean, some of them are certainly visible, but they're not, certainly they're not software visible like a lot of these. These things affect the manufacturability of the rack. They affect maybe the serviceability of some elements. I would love to talk about the screw elimination. Are we going to talk about that, Elliot? I'm not sure. I think that's on the agenda. Is that on the agenda? Yeah, where's that in the sequencing here? Because I thought that was a, and I would just like to emphasize, and this will come up a couple of times in this conversation, we have burned down zero data centers. I just want to emphasize that. I don't feel I'm not going to knock on wood because we're not... I don't know that I would have said that. Which way you would not... Do you feel we have burned down data centers? No, the tempting fate. The tempting fate. I knocked on the wood. That's good. Yeah, I'm now realizing that the absolute, the hubris and arrogance of me saying I would not knock on wood, which is, I mean, the gods listen for that. They listen for me saying that, and they're like, they feel that I'm taunting them with their lack of creativity. So we opened it with Pandora's box and now we're here. So, you know, and now we're here. Exactly. To address the question that came in about variants, I think that is actually a good question. We normally like just with a PEM change. No, it's pretty, it's pretty blind to all things. If we were cutting in just that change, we just upropped the part. Good to go. totally fine. With the sum total of changes that we're calling RM1 or like Rack Model 1, we don't need to worry about variance between Rack Model 0 and Rack Model 1 because it's such a large set of changes that we're actually, we're doing a compatibility break. Like there are, we will build Rack Model 1s as a separate, like not external facing product, but internally from like our design cycle, our data management, like we're considering it a new product. And so we've manufactured, on its own line as its own separate thing, distinct from Rack Model Zero, which is what we're selling today. Otherwise, yes, this would be an intercompatibility where we'd have a lot of spreadsheets. Yeah, but I mean, even in terms of compatibility, we are still using the same sleds. We're using the same sidecar compatible with the fiber patch panel. It's the same height, same doors, et cetera. So all of that, all the customer facing things that we need to have compatible, We've just defined our system boundary of the things that we're changing inside of that. Yeah, and so – and then occasionally, I mean, I think that, you know, when we shipped the first rack, we knew that the hardware is much harder to update in the field, to upgrade in the field. The hardware has got to be right, and then the software is going to have to – and then we need the software to be updatable. And so go back to our discussion with Dave Pacheco and team on what we did around software update. Just to give you an excuse, Adam, to both ring the chime and then it feels like the internet always has no opinion on the chime. Like they like it, they don't like it, and I want everyone to feel included. Yeah, that's good. But we did have a mechanical issue where we wanted to go kind of update it in the field. Doug, do you want to describe how we got here with the power adventure? Certainly. Yeah. So this is an instance where where in an opportunity in the field informed us that we needed to make a change to a certain aspect of the design. And we took the opportunity to make other improvements along the way. So as we, you know, the changes that we've been discussing here, as we go through those, we have to be fairly efficient with how we fold in changes as change is not terribly easy with existing product, with product in process on the manufacturing floor and so on. So we've got with any of these changes that we're suggesting, they have to be pretty calculated in when and how we make them. So in this instance, we had a failure, which drove us to make a change to how we – a safety aspect of the design, I'll say. And along with those changes, we found many other changes that we would like to make along with the same change. So are we ready for the photos? Yeah, we're ready. I mean, with that kind of lead in. We had an issue in the field where there was a cable. So on the back of our cubbies, our cubbies attach to our bus bar. And from the DC bus bar, we have two sleds. So they need a cable, a Y cable, essentially, to divert power from the bus bar to the individual sleds. So those look like this. All right. So case in point here on my part of lack of engineering rigor, normally when you tie off any cable, especially a power cable, you would want two means of protection here. So I don't think we're necessarily constrained by that in our safety standards for rack equipment. But from, you know, past history in the medical device industry, you always want two means of protection on your cable. So what do you mean by two means of protection? You currently see here in between what would be the conductor on the inside of the jacketing here, you have the live wires. You would want a jacket, which you see here, the red, you know, an insulation layer on the wire, the red and the black here, and something additional between it and metal as, you know, various mechanical conditions can cause that insulation to be damaged either at install or at, you know, frankly, through shock and vibration in use. You will see a case here. This was actually from our production floor, thankfully not anywhere further. Spicy photo here. What happens when there's a lapse, you know, a gap in that insulation? So in this case, we had a very high-power cable that contacted some metal. The metal cable tie insert there, you can see the square thing, nicked the insulation layer. and, yeah, it blew a pretty large hole in the sheet metal. Was this Robert? I'm trying to remember who was Robert. Yes, it was Robert. Yes. Ring the chime for our episode on holistic engineering. Robert really likes to go end-to-end, and I think as he did, I think – Doug, were you actually physically there? This is, again, as you say, this is unfortunately in our own facility. I was, yeah. It was my understanding that it was – there was a noise associated with this one. I was a little bit further away, but, yes, there was a noise, and I witnessed the aftermath. So, yes, this is not something we want, certainly. So, again, going along with, you know, finding things as we scale from 1 to 10 to, you know, 100 racks, this was not something we saw in our first mini racks because, you know, frankly, you know, with low volumes, It's a lot slower process to build these, and you go through a lot, you know. I think more care is probably taken at lower volumes as you have more time to build things. So as we're ramping up production and firing through things, yeah, go ahead. And it's also, I mean, this is in the cubby. So this is not something, I mean, you were saying earlier about, like, there's no moving parts. Like, this is something that you would kind of expect to be pretty inert. Like, you wouldn't expect something to kind of rub up against this. I mean, you would – or you can see how, like, we wouldn't see something like this, that it would take a while before we ended up hitting something like this. Correct. I thought that it could be good when you packed it up, and it goes through a truck and shipping. However, you get it from one place to another. It vibrates a bunch, and that insulation that was formerly together is no longer together. Right. that's also really important because i think that the uh the shock when you are transporting the the rack and we talked about the you know all the challenges operationally and and engineering the crate and so on but the shock and like how much shock and vibe do you have in transmission well it's like you hit a pothole if you're on a truck and hit a pothole that is a lot of i mean i we're talking about this earlier but what are the d forces that you get on a pothole typically around six g's you you tend to plan for nine if you're being conservative so quite a few that's a lot of g's that um so the lots of things gonna potholes can do i mean god only knows how much damage to computing equipment our our lovely city of oakland has done here with the it's famous potholes um or or insert insert your favorite city name i don't think i think that that's the oakland has no monopoly on potholes um the so and so doug this was as you you kind of took this opportunity to be like, okay, this is obviously bad, but we now, we've got an opportunity here to understand and improve many different aspects of this. Yeah. So, I mean, first we need an additional layer of protection on that cable, which again, you know, was a miss. Should have seen that right away. So we've got, you know, currently working through production, we add tape. The longer term plan here is just a plastic clamp to go around this. So no more, you know, cables directly on metal. There's an extra layer of protection there. So that's, you know, the minimum change we need to make here. But since we're making changes and in, you know, with the opportunity to drive other change and other opportunities that presented themselves as we were going through and reworking these cables, these cables, we found the way in which the cubby was attached to the rack left room for improvement. So working with our, you know, our field service team and, you know, our manufacturing folks as we made these changes, again, you kind of listen to what the pain points are of field service. And that's where, you know, we kind of discovered another issue that, again, it certainly was less visible than a big hole in the metal, but I'll show a picture of the back of the rack here. So you can kind of see in the bottom right here, these cubbies are attached to the back of our rack with two bolts apiece. In the center is our bus bar. Kind of on the bottom right there, to the right, I guess just above the sidecar fan and to the right, there's a bolt there that's pretty hard to get to I put it there It my fault But you need a very stubby little wrench to get in there and there hardly room to turn it And in going through and kind of you know reworking some of these units we really felt the pain of that both you know personally and with other members of the support team So the plan there was, you know, we're making changes to our design and manufacturing process. Let's take this opportunity. Like, let's change other things as well to make this better. So that's where, you know, we've gone through on our rack model, OneDesign added, you know, we've gone through and removed that bolt that was hidden, you know, behind the other. And this one is much more accessible, accessible for a power tool. And then, again, we can't just make changes like that just off the cuff. This is where, you know, the engineering rear comes in. And that's where, you know, Elliot helped prove that this was an acceptable change for our rack. And to be clear, because we're going from two bolts to one here. So, yeah, right. So this is we're going to be potentially closer to the wind here, closer to the margins. So we really want to understand what the mechanical consequences were. So, Elliot, how did you model that? What does that look like? And Brian has actually railed on me for this, for saying that it's not any more complicated than undergraduate statics class. First of all, to not rail on you. I just want to be very clear about that. But the remark that I did note, your remark of note was when you were demoing this to the company. You said, recall from your undergraduate statics class. And that was, you know, I think that it was like, well, you know, only a small fraction of the company may have been nodding along at that point. I mean, the double E's all took statics. The double E's all took statics. The rest of us are computer science concentrators, so we didn't have to take statics. That was the, you know. But anyway, yes, sorry. Yeah, but just a relatively – Recall from your undergraduate statics class. Exactly. Just a relatively simple load analysis. You take the bolt clamping force and you subject it to a certain amount of acceleration in the correct direction. Say acceleration due to a pothole because your rack is on a truck, for example. And you eventually get an answer that says, oh, well, actually, if we use an M6 bolt back here, a single M6 bolt, that's okay. But actually an M5 bolt is not okay. And those are the kinds of answers that we did get in this case and told us that we could actually go down to one bolt relatively safely. And that's like another aspect of just general precision machine design is the way I think about it. And what I personally find interesting about precision machine design is that it's all about answering the question of how do we make it work the first time and how do we make it work every time after that? And the analysis that we are justified in doing when we're manufacturing at scale like this, even though it is a box that does not really move, those are still very interesting problems that are present in the oxide rack. Yeah, absolutely. And then we changed the bolt as well, didn't we? Did we make it a slightly larger bolt? That one has always been M6. That has always been M6. Okay, so really just being able to go closer and proving to ourselves that we can actually do this in a way that will not collapse when we hit a pothole. That's right. The analysis showed that we could not reduce this to an M5. But it's like we do one M6, we cannot do one M5. Right. are you sweating a little bit when you're doing that analysis Elliot? I mean it just feels like this is like such higher stakes than software in that like, God, I hope I, we did a house renovation and now I am tempting the gods. Gods, if you can hear me right now, I'm not actually tempting you. We did a house renovation and we were going to have a window and our structural engineers, like you can't have a window there because I've done the structural analysis and in the event of a seismic event, the moment frame would be, what was the language they used? And so we're like, great, we'll eliminate the window. And then they came back later, they're like, actually, you can have the window there. And I'm like, okay. I double-checked the math. It's actually the window is fine. You're like, is it fine? Now I, like, and it's like a window in a kid's bedroom too. And it's like the kid always has like the blind closed on it. I'm like, can you at least open the blind in this? If this window is going to result in this house collapsing on itself, because I'm just worried that the updated calculation is wrong. I know this is just like this is just part of being an engineer, but just the consequences feel high. Yeah, that's the point of review. That's why we have a nice mechanical engineering team that can review those sorts of things, check the math. And then there's always the mechanical engineering fudge factor of factor of safety. Yeah, right. Which is basically how sure do you think you are? How much uncertainty is there? Yeah, interesting. In aerospace, the factor of safety is typically around 1.05 or 5% more than exactly what you need. Or 1.1 times. Yeah, not 2x. Not 2x. No, no. But in agriculture, which is where I was coming from directly, it was more like two is the minimum that you're working with is typically more like six or eight. Really? In agriculture. In agriculture. And principally because, I mean, the safety consequences are pretty grave. The safety consequences and there are a lot more uncertainties that you might face in the use applications. You're right. Meaning people will use the things that you engineer for almost anything. Also different. Do you remember when we. Yeah, sorry, Brooks. Go ahead. Go ahead. Well, I was just going to say that on the – do you remember when we had that small cluster of earthquakes when we were living – when we were at the Fishworks building? And we all like – and we're all like, okay, we get it. We're going to die in this building that was built in 1920. And we were talking to – we had a contractor doing work at the time, and he's – and we were all confided in them. We had now, like, resigned ourselves to die in this building above Walgreens at first submission. And he's like, no, no, actually, this building is great because it was built in 1920. the city had just collapsed and burned within very recent memory. Nobody trusted anyone else's estimates for anything. And everybody was doubling everyone else's estimates. So you are in this era of unbelievably over-engineered buildings in San Francisco. And it sounds like this conversation stuck with you as well. You could drive a Buick onto this floor and it would not. He's like, the building to worry about. And then he points out a building across the street that was brand new. He's like, that's the one to worry about because that one is built right up to the code. Now, I'm not sure. I was still relieved to move out of that building, Adam. I don't know about you. I just love that this is the same engineer who used like decorative two by fours here and there in his HVAC work for us. Yes. So, yes. Yeah. Engineer may be in air quotes on that one. But just in terms of like understanding kind of where those margins are and engineering around them and then being obviously very cautious. when you are. Yeah, but also not overly cautious. Like if we were going to aerospace standards on everything, we would never ship the rack or we would ship the rack in a couple of years after the rigorous engineering analysis. So being able to do the correct amount of analysis is also a critical part of this job. And also what is interesting about mechanical engineering generally. Yeah. And Doug, can you talk a little bit about the high pot testing that we did based on this? Because the other thing that I really appreciated about that, when we saw this problem, we're like, okay, let's immediately develop a test where we can find this thing before. Because we obviously don't want to ship this to any customer. Like, how would we test on the manufacturing line? Yeah, absolutely. So I probably won't do this. The dude, Justice, that I think Nathaniel's on or somebody else. One of our electrical engineers will do, but I can speak through it the best that I can. So basically to find, you know, issues with compromised insulation in cables, there's a high potential test that happens to, I don't know, run some kind of electrical magic through the cables in the rack to determine where there is a potential gap less than we would like or dielectric strength rather less than we would like between, say, a live wire. and a piece of sheet metal. So we have a tester, Eric Austin developed a tester that plugs into our rack into the regular sled spot that he hooks up with a, that he connects to a high pot tester. And I want to say it's 2000 volts. It might be more than that. It's tested far above what our rack would normally see in the field, well above, to identify issues, potential issues with shorts and insulation. So this was a test that was previously done on a finished rack. We have since moved it to be done on a cabled rack that does not yet have sleds to find issues earlier. And when we find those issues, the cubbies can be pulled out and analyzed to see where the potential lapse in insulation may be. Yeah, which is great. I mean, I kind of love that feeding back into the manufacturing process. Like, okay, how do we make sure we don't, now that we know this problem can exist, how would we go find it earlier? And so then with that issue, and then so after we were able to prove to ourselves that like, we know we, one, can't use the M5s, got to use the M6s, but we can use one M6. And then we know that we don't ever anticipate having to do this in the field again. But if we had to, it would be less work. But importantly, it's also less work to manufacture. It's another one of these things where it's like, Now it's a faster manufacturer. It's got less parts to it. I mean, it has all these knock-on effects. Yeah, so exactly. So all the pain points that we saw of having to access that bolt and attach it are gone now. So they're much easier to attach at our manufacturing partners and much easier to service in the field. So, and then on manufacturing, so, you know, the thing that you and Doug, I mean, I love this with the, it was eye-opening for me when we were doing, you know, building a wholly new product. And one of the things that you did is we 3D printed a lot of parts, which allowed us to really iterate quickly on what we want to go build. But now we know what we want to go build, and we want to now do it at greater scale. Like 3D printing breaks down a little bit at scale. There are cost issues and other issues. I mean, Brooks, you came from Formlabs. You did 3D printing and had a kind of great insight there. And I know, Ben, you were really kind of at the forefront of kind of replacing some of these printed parts from molded parts. Can you kind of go through the calculus there, Brooks, about how we think about 3D printing and manufacturing? Sure, yeah. I think this is mostly going to be a Ben answer, but I can tee him up a little bit. Like you said, printing is great for figuring out what you want to make. I'm going to say cheap, and then I'm going to say it's expensive, but it's cheap, it's easy, it's fast, you can just do it. Boom, one, you know, you got the parts in a week, and you're done. or you have one in your lab or your house or whatever, and you have parts in an hour. But the part per part cost is massively higher. So one of the things, and I'll hand it off to Ben at this point, one of the things Ben has been doing is figuring out everywhere we can possibly injection mold parts instead of printing them or machining them or whatever. So over to Ben. Yeah. Thanks, Brooks. I have spent quite a bit of time after joining Oxide focusing on converting 3D printed parts to injection molding. It's really a very big geometry challenge. Injection molding has very different requirements from 3D printing. Say more about it. What do you mean? Yeah. So generally with injection molding, we're trying to basically have two halves of a mold that we can squish together and then plastic is injected into the mold and then the mold pulls apart. So in order to do that, we need to make sure that the parts kind of like all coming off in the same direction. When we 3D print a part, we can kind of just have whatever shape we want to an extent, making sure it's still structurally sound, but it can have a bunch of holes in it all sorts of different ways. We can keep the material thicknesses different sizes, but with an injection molded part, you need a wall thicknesses to all be the same thickness, more or less. And then we also have to worry about what are called draft angles. So say if you had a cube with a kind of open face on it and you tried to pull that off of a mold, if it was just a flat wall, it would be pretty hard to pull it off, right? You get a lot of drag. But if you draft it, so you just add a few degrees to your walls, now it's a lot easier to pull that part off. so generally the exercise i'm doing is taking parts that we already know work and they're 3d printed but they're not going to work for long because we need to make lots and lots of them and it's just not very feasible to keep 3d printing them so we need to and that is mainly a time issue more than a cost issue i assume yeah interesting yeah yeah we can buy them um they will be very expensive, but we're mostly focused on making sure we can get them in time is the main constraint right now. So what are some of the examples of parts that you kind of got through this process with? For sure. I think I have an image of one. There we go. So we've got these transceiver blanks. So we've got our switch on the front of the rack that's got ports for a large amount of transceivers. I haven't memorized a number yet. We don't usually populate all of them. Usually we just have a few transceivers going into our switch. But we don't want to just leave an open hole there. 64. Thanks, folks. So instead, we're putting a 3D printed kind of like plug in there. And then also the fiber will go into that plug to kind of protect the fiber and keep it routed where it's supposed to be. So pretty simple function, but there's a lot of complexity there. We need to make sure it mates properly with two different parts. And we have to kind of meet some fiber standards to make sure we're doing that properly. So there's a 3D printed version of that that's working quite well. But I kind of did a comparison on the left there. And then in that picture, there's an injection molded version of it on the right. I've kind of added some details on it to make it, you know, a little more aesthetically interesting. But also there's just this big challenge of taking kind of a big chunk of plastic and coring it out in a way that's pleasing to look at and is also possible to pull apart in a mold. Yeah, interesting. And then is there a – are there tools to kind of help you do this in terms of where you need to change the geometry? I mean, I feel like this has got to be a pretty common journey for a part where it starts off where you're printed at low quantity and moves to injected molding at a higher quantity. Is that – or is this – yeah, interesting. Yeah, I mean, generally we're just using SolidWorks' built-in tools to do this. Yeah, interesting. You can look at like draft angles. There's a tool to just kind of visually see. I don't have an image of it, but you would have areas in red that are one direction and areas in green that are another direction. It would kind of tell you just visually real quick, am I am I able to pull this part in the direction that I want to? And then you can also do things like checking for wall thicknesses, make sure, you know, all your walls are are an even thickness. And generally, we can work with our molders as well. They'll provide a level of kind of rigor that isn't quite available to us just with our CAD tools, but they'll do some plastic simulation to kind of show the quality of the mold. And we can take that feedback and kind of iterate a little more before we actually create the mold. How is the mold made? What is the mold made out of? um depends a bit on volume uh most of my career we've been doing aluminum molds they're they're cheaper um but they don't last quite as long uh we're actually doing a few steel molds in this case because our volumes are are just that high so it doesn't really change much from my perspective about designing the part but i i think from the molder's perspective it is different and is that Is that like milled then? Is that how that is manufactured? Is that CNC milled? I mean, how is that thing manufactured? Yeah, it's usually CNC'd. In some cases, they'll use wire EDM to do especially difficult features, right? They just have to make it one time. So it can be a pretty expensive process to make the mold. But yeah, you can get into some pretty sophisticated machining technologies there. Yeah, wild. And then I feel like we've joined the big leagues that we're actually going to steal molds for things. We've really arrived, grown out of aluminum molds. And I mean these parts – and I don't know if this is an example where you use steel molds. But there are some of these parts we're just going to make – we make a lot of them. Yes. Yes, the volumes are kind of like talking with our supplier. And they're like, that's annually? Yeah. Yeah, I guess it is. Yeah, wow. I mean, because I think we always think about terms like rack volume, and the rack volumes are lower because they've got – but the rack contains many, many, many thousands of parts if you include all the electronics. So it's like a small amount of rack volume can actually have a lot of parts, and a lot of rack volume has many, many, many parts. The highest quantity custom part in a rack is possibly this part we're showing on screen, but it's up to 320 of a specific part inside the rack. And, Brooks, what is this part that we're showing on screen? Ben, you want to take that one? Sure, yeah. So this is what we're calling our SSD blank. we're kind of looking at having SSD cages that don't actually have an SSD inside of them. They're just a piece of plastic instead. In cases where you might only need a couple SSDs on your server, we still want to fill in the rest of the server to make sure we're still have the same thermal conditions and EMI conditions and all that um so yeah this is a kind of just yeah rectangular chunk of plastic that's that's doing that function so you i mean you raise an important point because this is not just these these blanks are not just for aesthetics but we've got these other and they um adam can ring the chime again for the oxide the chamber of mystery he's talking about like all for electromagnetic interference and when you get compliance in the rack, it's like you can't just then ship a rack with all the stuff blank, especially for EMI. But you see the thermals as well. So everything has to be engineered. Absolutely. This is actually one of those changes you do so you don't have to do as much engineering We don know that you don need drives in there to pass EMI testing or thermal testing or all that jazz But we know if you put the drive in there it passes Right. And we don't want to redo safety testing and EMI testing because it's time-consuming, expensive, and painful in whatever order, depending on how you're feeling that day. and so and so making this change and putting this part in there prevents us from having to go back and do all of that engineering to prove or testing to prove that it's still safe compliant whatever yeah yeah interesting and so and this is our highest quantity thing in the rack uh especially those that i've got some folks that want to have different balances of storage versus compute and so on. So they end up with a lot of blanks, eventually. And I do want to touch on cost, actually, on this part, because I think this is one where it does factor in in a major way. Obviously, this is a lot cheaper than an SSD, or orders of magnitude, in fact. But the cost trade-off between 3D printing and this is where the 3D printing is cheap but expensive comes in, versus injection molding. When you're printing, say, this part, This part would cost 50 bucks to get printed if you got it out of equivalent materials to what we're molding them out of, which is fine if you need 10. But if you need 320 per rack, that's actually a lot of money very quickly to get the mold for this part. Is it going to be I think this one's like 50 or 60 thousand dollars, which is on the expensive side for molds. And feel free to correct me if that number is wrong. At least one of our molds is up. Some of our molds are up in the 50, 60 thousand dollar range. So obviously we really need to know that it's right before we do that because that's expensive. But now every time we buy one of these blanks, it's like $2 or $3, $4 maybe for a larger injection molded part. So there's a lot of upfront tooling cost and then very, very little cost moving forward. Yeah, so this is one of those things where when you get to new levels of scales, you were talking about problems that kind of ones versus tens versus hundreds versus thousands. you get to a crossover point where it definitely makes sense. Yeah, and Ben actually did graph that crossover point, and it's surprisingly low quantity for some of these parts, actually. Interesting. It makes sense. One of those parts is hilariously fast payoff. I'll share it sometime. It's when we have a blank for the whole sled. so if we're not completely populated with servers that 3D printed part is immensely expensive so you pay for it extremely quickly so is that one where you hit the cutoff point earlier not because of the quantity of part but because of the complexity of the print yeah and just the volume of the print really prints get very expensive once you talk about larger parts because you kind of have to print the entire volume of that area. So it's a lot of time and it's a lot of plastic. Yeah. And Doug, you know, I still have, correct me if I'm wrong, but the very first air shroud you did was 3D printed. Yes. The first gimlet, that gray shroud. And I did not realize that you could, like, I just didn't realize what you could 3D print. And it's like, well, this is because it's very robust. It does not feel, you know, it's a long way from a home 3D printer. Let's put it that way. Yeah, those were $600 each. And now at scale, we're molding them for less than $10. So, yeah, there's the cost difference there. Yeah, and we did – Brooks, do you want to describe the – because we have a clear shroud on Cosmo, which is very exciting. Yeah, much to Brian Chagrin. You know, no, no, no. Not at all. You've come around? I've come to love the clear shroud. And you know what? I have come to love that you love it. That's actually the more important thing. No, I do. I have come to the clear shroud is pretty cool. Do you want to describe how we got there? Yeah. So on Gimlet, the airflow shrouds, do we have a picture of the airflow shroud we can pull up for context? We have an airflow shroud in the middle of the sled that makes sure all the air goes over the DDR5 or DDR4 and the heat sinks. and all that jazz and doesn't just pass uselessly over open air because thermal efficiency is good. That's a big injection mold apart. That's one of those $50,000 molds, $10 apart. It's kind of a lot. But it's fundamentally like a two-dimensional sheet that has been bent and shaped, a lot like sheet metal, but plastic. And so one of the things we looked at pretty hard for Gimlet, or for, sorry, for Cosmo a little while ago was if we could make this even cheaper than injection molding. Because we're going to be making a bunch of these things and, you know, cost savings do add up. And it's a kind of silly injection molded part. It's huge. It's very, very wide. So we were looking at a process called thermoforming, where you basically, you machine out of wood or steel or aluminum, kind of any rigid, thermally stable material. you machine out the inside of your shape, whatever that might be. So if you were going to mold a drinking glass this way, that's a terrible example. If you were going to mold a plate this way or a bowl, you would machine exactly the piece that fits inside the bowl and you would flip it upside down and then you'd get a sheet of plastic, like ABS in this case, really, really hot, not melted, just up to the point where it's sagging, it's soft, it's becoming gooey. and you smash it down over this mold and you pull a vacuum on the whole assembly and it sucks down and it forms exactly to that uh that shape that this is how um if you get a salad a salad bar that's how those are made that's a thermoformed clamshell piece or like oh interesting that's all thermoformed uh because the again mold costs something but then at scale the parts cost nothing like they're almost free at that scale for that kind of part um it's like penny pennies per part and the thermoform shroud was going to be like a cut like three or four dollars per part and like twenty thousand dollars of mold so from a cost perspective it made a lot of sense and we and in so doing we're also looking at oh could we make this clear could we um and kind of get a, as it's been posted in chat now, kind of like the clear Game Boy Advance style, like translucent purple or translucent or just like fully clear aesthetic so that people can see into the sled when they're looking at it. Kind of for like a marketing perspective of like, hey, look, here's your computer. You can see your computer. Isn't that cool? And because we all grew up with transparent electronics in the 90s and we just want our purple Game Boy back. So we were looking at, like, can we make a green one? Can we make a clear one? What are we going to do? And the Thermoform parts looked bad. They looked like they were ugly. They did work. They worked fine. But, oh, buddy, they were not pretty. And if the point was to make this a cosmetic part, that's not the point. So we would have been saving a lot of money, but like severely degrading the aesthetic quality of the product. Yeah. For something that we're doing in order to make a cool window into the technology, it doesn't make sense. Right. But at this point, we kind of weaseled the idea of a clear airflow shroud into the oxide consciousness. Totally accidentally, totally not deliberately, if anyone asks. so when we switched back to injection molding and did that major update on that part for for cosmo we did the first shots of that tool in clear because it's easy to go from clear to opaque and harder to go the other way because of mold texturing that we don't need to get into but so we got the first shots of that airflow shroud in clear kind of without asking and then just showed up to Oxcon with them and were like, look, isn't this cool? As a little internal marketing pitch. And they're still clear. So it worked. It definitely worked. No, they look pretty good. I mean, I love the old Trouts too, but I love all my children. Don't pick your favorites, exactly. Exactly, but those clear Trouts look awfully good. And they definitely look better than the third form ones. It's actually a relief to know that you had the same conclusion. Yeah, it was not good. These are not good. Hey, Brooks, the other day you were demoing the manufacturing process or some of the changes that you were making in Rack Model 1 explicitly to make it easier to fabricate. So I was wondering if you're going to talk about that some. And then I was also wondering about the degree to which we collaborate with manufacturers on design. You know, back to Brian's home renovation example. You know, I talked to a general contractor and then we did what was called value engineering. Like, how do we take these plans and make it so we can actually build it and you can actually pay for it? So I'd love to hear about, you know, working with manufacturing to make these fabricatable. Yeah, I mean, I could talk about that on a house or on a rack right now. I have carpenters in my basement this morning replacing all my beams. so it's a close to close to my heart example um but yeah no we do we work with our our uh supply partners a ton um many many of the individual specific changes between rack model zero and rack model one came out of a conversation that uh i believe doug and ben had at one of our supply at one of our like key suppliers uh with their their engineering team their um their quality managers their um they're like people from the factory floor who are building the racks all got in a room and just talked through every single thing they had on Rack Model Zero that was, hey, this is kind of annoying. This is hard to fix. This is hard to do. This process thing here has rough edges. How do we literally or figuratively, like the PEMs came out of that conversation. It was kind of on our radar, but they were coming in and saying, hey, these PEMs are a huge time and cost problem. And they're really annoying for us. And they're really like messing up our workflow. can we get rid of them? And we're like, I don't know. That's fair. Figure it out. We'll try it. And then it became like one of the major changes. So we're constantly engaging in that kind of conversation with basically every supplier we work with. Like we're not, and I think this is true at kind of any scale of company, assuming you have good suppliers. Like Ben's been having tons of back and forth with our preferred molding vendor, or one of our for molding vendors about all these injection molded parts he's been doing. They've gotten a lot better through that process. We can do a pretty good job in a vacuum of looking at the rack and saying, okay, here's the things we need to change to make it more wonderful. But they're the ones really living it. They're the ones who have to touch the stuff every day. So that's one of our most valuable sources of feedback is our suppliers. Well, and we really want our – we have got a design that's hard for them to manufacture. We want to hear about that. We want them to give us that feedback. So that's – and Brooks, you want to talk about this kind of the specific example that Adam's referring to? Because I think this is a fascinating example of something that is effectively in the product but is not relevant from like the day the product ships. by the time the product goes into the crate, this aspect of the product is no longer needed. Oh, yeah, yeah, yeah. It's very important for manufacturing. I missed the tee-up on that one. That's a good... Thanks for bringing me back to that. So right now... I assume you're talking about the welding. So right now, the rack is welded together. And again, we'll go back to the several massive chunks of metal. There's a big, heavy metal base. There's four big, heavy metal verticals. There's a mid-shelf. There's a top plate. There's like a roof. and a bunch of other ancillary stuff. Right now, our sheet metal supplier has one big, like, endoskeletal contraption and a bunch of clamps and magnets and stuff, bits and bobs. It's a whole kit that holds all of those pieces together precisely enough that they can then be welded in place, and that all gets taken off. And that takes somewhere between half a day and a day, depending on who's doing it. how well it's all going per rack. And again, if we're talking about making more and more racks and making this process more efficient, a process step that takes four to eight hours of more than one person's time is a really good place to look. So Rack Model 1 has some very simple modifications. And I don't have any good pictures of this because it's not visually interesting, but it's technically very interesting. What we've done is that all of those joints, we've gone in and we've just slapped a couple rivets on there. And this kind of goes back to the engineering analysis end of things. We could completely redesign the rack to not have any welds in it and to be all riveted. But one of the things about the oxide rack is that it looks very clean, polished, unified, and you don't really get that with rivets. Rivets look more tacked together, in my experience. It looks bolted together, you would say? Exactly, it looks bolted together. And so we want to stick to that like kind of uni, like uni body, single part. This rack is a thing. It is. This is one object, both aesthetically and structurally, because it is it's very heavy. There's a lot of structural analysis that we need to go into changing the welding because it's like thousands of pounds. So what we've done is we've gone and we've added a couple simple rivets at every single interface there. So instead of having to take this rack and like jigger it into this whole contraption, they can just line a piece up and bang, slap two rivets in, put the next piece on, bang, bang, slap two rivets in. And instead of it being several people taking a lot of time to carefully align all of this, that alignment is built into the geometry of the metal itself. And then once those rivets are in, they can just kind of step back and start welding and they're free to do so. and then that too is nice because we don't have to go build more fixtures if we want to build more racks we just need another welder right it's much easier to go to i mean not harbor freight but like it's much easier to go to your welding supply store down the street which they have uh don't we all and just get another welder if you need to add more capacity to your line versus like tens of thousands of dollars and hundreds of hours of engineering time to add another welding station I love this because this is a very – this is like as things go, this is not a complicated or certainly not a high-risk change to the rack. And what it ends up – but you're kind of adding this thing to the rack that then solely for the purpose of it being more readily manufactured because it's the weld ultimately that holds the rack together, not the rivet. Yeah, the rivets are barely structural. we've specifically asked them like, Hey, are you going to move the racks around after you weld them, after you rivet them before you weld them? And they were like, no, no, no, no, no. We're not going to do that. Like, okay, good. Because it, they're not that, they're not that strong. Don't do that. This is not rated for any pothole or for that matter, any movement really. So this is, we do not build these in a seismic zone and that is good. Right. This is like, I would say this is like a Kia furniture that I've assembled. That seems to be held on with kind of a, a single whatever those little wooden goobers they call the wooden dowel pens yeah okay yeah yeah right um but this change to like we're saying it takes four hours to weld or act together like this change took maybe 12 hours of engineering time like start to finish yeah like started on a monday morning and it was done like tuesday afternoon with other stuff happening in the meantime so it's not it's not a complicated to change, but it's very impactful. It's not a complicated change, but also it's a change that you have to know to make. And you know to make it because we've got this kind of end-to-end visibility of this thing from when it is born as sheet metal to all the way to hitting the potholes and all the way into a customer data center. Yeah. And there are a lot of people that will say their suppliers are good because they'll do literally anything you ask for. And our suppliers are good because they will do some of that and they will also ask us for things that will make their lives easier yeah that sounds great and if we ask them to do something too crazy but cock an eyebrow at us and say no if you want to like we'll we'll we'll we'll bill you for that but are you sure i mean that's well how i why i view it through the lens of a general contractor where you want them to say like you really don't want to pay for that like that's i hear you asking for that, but I'm going to do something different. Right. Like I can technically do that, but you know, it is not in your economic self-interest that I do that. Exactly. Uh, yeah, this is, um, I had another contractor with a, I had an architect back in mathematical error and, um, the, that the contractor spot is like, I don't understand why we're zigzagging this wall. Like I can just make this straight. It will be much less concrete and we'll open up square footage in your house. So what are we doing here? That was a very impactful change, as it turns out. Open up 300 square feet. That's where my Cosmo is right now, that little basement lab, thanks to a contractor that was willing to push back on the architect and get to a better design. Well, this is awesome. I feel like, I mean, we could talk for hours here. I think we may have to have you all back to do a second episode here because I feel like there's a lot more we can talk about. But I do want to be mindful. Adam, I know you've got Little League to go coach. Is that right? Little League. Exactly. Big game. God, that is so exciting. Are we going to get your Little League team on here for an episode? We've got to – I mean, it's baseball season. It's exciting. Sure. We'll do a live broadcast. I'll do sound effects for the hits if there are any. You know, that is my dream to do, Adam. I want us to do play-by-play and color commentary for baseball together. I think we should do an Oxide and Friends where you and I, we're just going to be at a ballgame. Stay tuned, everyone. Exactly. Make sure you subscribe for that one. Right. This is one of my ideas when Adam is like, are you worried that we have too many subscribers? Is that the problem we're solving for now? There's simpler solutions, right? It makes sense to me. Let's get rid of those folks. It's been awesome. And Doug, Brooks, Elliot, Ben, thank you very, very much for, I think, again, we're going to have to have you back because we've got a lot more we can talk about mechanically. And there is just so much here. I feel like I must feel the same way. I just did not. I don't look at the rack the same way. I mean, there's so much about building this product that looks simple and is just not simple. There's a lot of terrific engineering in it. And thank you all for all the terrific engineering work that you've done and keeping it safe. Safe but not too safe, right? Let's not get that safety margin too high. Thank you for having us on. This was a lot of fun. A lot of fun. Happy to come back anytime. We have tons more to talk about. We didn't even touch on any of the R&D work. This is all just on the manufacturing end. All just the manufacturing. We are expanding the team. So if you are a mechanical engineer with a similar disposition, and yeah, we are not just doing manufacturing work. That's been a lot of what we're doing, but we've got a lot of exciting work ahead of us as well. awesome all right thanks everyone adam go get to that little league team um uh always send the runner to second i trust i trust you're at that stage where uh excellent free base over there um and uh we will see you all next time