自建数据中心：算力成本核算与训练推理分层策略

You said 11 raps kind of leaprogged you.

Is that on you?

Yeah, 100% it's on me. It's the biggest strategic mistake I made in the history of Speedify.

How do you reflect on that?

So today is a real freaking discussion. Cliff Whitesman, founder and CEO at Speechify, one of the fastest growing text to speech startups in the world on the show.

The best way to lose is not to be in the race. Be in the race. You don't want to be a fat manager who is like a general sitting in the back saying, "Take that hill." You want to be the warrior who runs up with their sword and engages the enemy first.

Ready to go.

Cliff, it is so good to have you back in the studio, dude. I I was looking forward to this one cuz when I was writing it up, it's a very different thread of conversation to how I'd normally go. And so, thank you so much for joining me again today, dude.

My pleasure. Glad to be here as always. Now, I wanted to start with you're spending tens of millions of dollars on Nvidia GPUs and you're paying an additional $100,000 per GPU to receive them 4 months early. Why? Like, what do you know that the market doesn't know? So, in 2022, we bought a huge rack of GPUs from Nvidia. And the reason we bought them is for training, right? We have a bunch of models. The newest Speedify Simba 3.2 2 model is ranked number one in the world for quality um above all the frontier labs 10x more affordable and stuff like 11 labs and we used to rent GPUs and we found that engineers at speechify would be parsimmonious with how they use the GPUs cuz they were like oh my god I'm costing the company tens of thousands of dollars like I don't want to do that and the analogy my brother and I came up with is imagine you're Michael Jordan and you want to be in the NBA it's the only thing you care about and you need to pay $20 an hour just to train in a basketball center well that sucks you want one that you can go to whenever ever you want to.

In fact, you want a hoop in your house. And so our initial idea was we want a hoop in our house. And so we bought a bunch of our own GPUs. And that deal ended up being really good for us. And we ended up training really good models. So with time we invested more and more and more and more. So that's the first part. The second part is actually how the economics work out. So if you look at it, the transformer was invented inside of Google in 2017.

Nvidia came out with A100 GPUs in 2019. Shortly after they came out with H100 GPUs, right? The original Chetchup PT was trained on A100s. And then they came out with Blackwells. So then B200s, B300's, and now they came out with Reubins, which is the GPUs that Elon is sending to space. And they're like liquid cool cooled. They're very, very cool. And we're like, okay, huh. One, every class of GPU is more affordable per one trillion flops, right?

So, a flop is addition, subtraction, multiplication, any mathematical operation, and you measure them in how many trillion of operations happen per second in a GPU. And so, they're more affordable as it relates to this. Um, if I was to buy an H100 for, let's say, $30,000, that's how much the kind of a single card would cost. If I wanted to rent an H100 for one hour spot instance from GCP, it could cost me $5. If I rented it from like, you know, Azure or AWS, maybe it'll cost me $3.5 per hour.

So, if I multiply that times 24 hours and then times 365 days in a year, I'm actually going to end up paying $35,000 to $50,000 to rent that GPU for one year, but I could buy it for $30,000. So, it's 1.5x the cost of owning the hardware to rent the hardware for a year. Now, the hardware is typically um warrantied for 3 years to work properly, but it'll keep working up to the warranty for I imagine, I don't know, 10 years.

So the math just maths where it makes way more sense to buy them. The other big part is if you want to do large scale training like we do, you need the memory to be colllocated with a large cluster of GPUs. I can't just rent from Google or Microsoft or even B 10 and run the size of training that I want because I need a gigantic memory card next to it with all of my data that all the GPUs are accessing. So that's why we first started buying them.

The next thing that we found is actually if you run open- source models for coding, you could pay anthropic and then you know you're paying for all the tokens and the fact that you're doing the the the branded right fable one or you can run an open source model and instead of running it on a spot instance from Azure or anyone else you run it on your own hardware and then you're paying a fraction of a fraction of a cent per token and so for all those reasons it made a ton of sense but we can go into all the depth that you want.

I just want to dig in. The first thought that I have is I completely understand the rationale there but chips depreciate. You have chip cycles and they are accelerating. We are seeing newer and newer chips being created. We're seeing specialization within chips by buying your locking yourself in so to speak to one chip architecture. How do you think about that? At speechify, we still use K80s for a lot of specific operations for inference and we use older models of GPUs constantly.

Um, and there's essentially a difference between when you do inference and when you do training. For training, I'm like, okay, I have this hypothesis. I want to know the answer to this hypothesis as soon as possible. Like every minute that it doesn't come out, I'm in competition with everybody else. And so having a GPU architecture that is much faster by orders of magnitude is a huge advantage. But if you go speech to text or text to speech with speechify, I can afford to give you a lower quality GPU and it'll give you what you need still in, you know, 100 milliseconds.

So it's like totally good. And so I can always use these older GPU models for inference. That's number one. Number two, we have so many experiments that were running at every single point in time. Not all of them need to run on like the newest hardware. So, the analogy I always give, let's say you bought an iPhone back in 2011 and it's an iPhone 3G and then you bought another iPhone and another iPhone and another iPhone.

You could have a drawer in your house with like five iPhones that are collecting dust cuz you can only use one iPhone at a time, but if I own a 100,000 GPUs, I'm still going to use all of them at the same time. And so, I'm not losing anything by having more GPUs because not only do I own a bunch, I still rent from the hyperscalers all the time. And I rent both dedicated instances that I prepaid for and I rent spot instances.

For example, more people use speify in September because everybody goes back to school. So I need to like level out the load. And so the parts of that load that I know for sure I'm always going to use whether it be training or it be inference, I might as well just own it. And then on top of that is also the case that I have so many other friends who are running training and running inference. I can always rent it out to other people if I have excess capacity, which I don't expect to have.

But like every once in a while you have an interesting situation. So for all those reasons, it just makes mathematical financial sense. Lastly, if you have excess capital really, you either stick it on the bank or you buy a bond, right? Like the best long year, the best bond you can buy long tail, I don't know, will yield you like 5%. Or you can buy a GPU and because renting it would cost me 1.5x buying it for the year, the return is like way higher.

So how many GPUs do you buy then?

So let's talk about Reuben's for example. So Reubins come in the form of 72 cards in one rack. So we'll buy multiple racks of Reubins and then on top of that we'll buy B300s which are like the newest form of Blackwells uh because we can get them earlier. And then the same thing like you know when we bought our f