Share from 0:00
0:00
/

🎙️ NEW EPISODE: Google Built Two New TPUs Without New Silicon: The 8i Max and 8TX

Google's TPU 8TX and 8i Max, confidential computing with AMD EPYC CPUs, TPU Sync for memory management, Boardfly vs. 3D Torus topologies, and more

Austin and Vik break down Google’s two new TPU systems, the 8TX and 8i Max. These aren’t new silicon, but rather new system configurations built around the existing V8 chips. They explore why Google is creating a secure version with an AMD CPU, a memory-maxed version for inference, and what the new TPU Sync software layer means for the ecosystem.

Presented by Crusoe

Fine-tune and serve open models without managing a single GPU.

Deploy on Crusoe in one click, or download and run it anywhere. Upload a dataset, pick an open model, and pay per token rather than per GPU hour.

Accounts are free, and get $5 in credit when you sign up.

Try Serverless Fine-Tuning →

Things we cover:

  • The new TPU 8TX and 8i Max systems

  • Why the 8i Max doubles down on memory for inference

  • Confidential computing and the role of AMD’s EPYC CPU

  • The mismatch: training silicon (8T) with an inference network (Boardfly)

  • TPU Sync, Google’s answer to Nvidia’s Dynamo

  • The business strategy of selling TPUs to external customers

This episode is supported by Meter.

You have the GPUs and the power. Now who builds the network, and who do you call when it breaks? Meter designs, deploys, and runs frontend, storage, and management networks on its own switches and its own network OS. One call reaches the team that designed the hardware and wrote the software.

See how Meter builds datacenter networks →

This podcast is lightly edited for clarity.

Four Flavors of TPU V8

Austin: Google just announced two new TPUs. Except, they really aren’t new chips. They’re just derivatives of their V8 chips announced earlier this year. So Google is taking existing silicon and building two very different systems around it. We’ll also discuss TPU sync, which looks a lot like Nvidia Dynamo. What is really Google up to here? Let’s get into it.

Hello listeners and welcome to another Semi Doped episode. I’m Austin Lyons of Chipstrat. I write a newsletter for institutional investors and technologists.

Vik: I’m Vik Sekar and I write a substack called Vik’s newsletter and I also am the founder of Semi Exponent, which is an independent research firm for institutional investors. And you should check out our daily newsletter on daily.semidoped.com. We cover all the daily news around the AI and semiconductor industry and that is an entirely free newsletter to read on Substack. So you’ll have that delivered every day to your inbox along with our takes on the news. So sign up for that at daily.semidoped.com.

Austin: Yes, and which by the way, I haven’t published ours today. I think you put your takes in there and you’re waiting on me, but we’re recording this on Thursday, November 12th, and we were going to record this TPU thing, but then of course, there was some big news this morning about Nvidia and Jensen Huang retiring. So instead, we just recorded a hot take and then so people will have already heard this. And now, in the same day, we’re wearing the same clothes, and it’s like almost midnight for Vik, and we’re recording this TPU one as well, which means I didn’t get the Semi Doped daily out yet. So let’s record this and then if I remember, I’ll try to add my takes and hit send.

Vik: Yes, the AI world never stops. Time zones do not matter. So we have to get to it. So, what’s with the whole Google thing? Let’s get into it actually because if you saw the announcement, Austin, it’s like suddenly there were two new TPU SKUs that landed. And the earlier ones announced this year for the version 8 TPU chips was the TPU V8T for training and the V8I for inference, which was a nice way to name it and it was very understandable. And we were very happy with it when we discussed the networking that went around these chips at the time, which was basically they introduced this new network topology called Boardfly, which we explained in a YouTube video, a podcast here. And it was seemingly one of our most watched podcasts. People really seem to like it. So anybody wants to check that out, should be there in our back catalog. But the recent announcement a few days ago is that we have two more versions of this eighth generation chip, which is the TPU 8TX and the TPU 8i Max. Okay, so we’re going to go and discuss what is happening with these four versions of the same chip in one generation, that is version 8. So what do you think is going on? What’s happening here?

Austin: Yes, yes, so first of all, I barely saw the news because I think this came out at the start of the week and obviously there’s all the Nvidia stuff in the news. And so most people’s eyes and attention was on the Nvidia stuff and so this was kind of as Cloud would say, it was quietly announced that Google had some extra chips here. And I was surprised. I was like, wait a minute, there’s four. Two was kind of a big deal because it used to be back in the day that for most all the generations, the TPU just had one and they use the same one for training and inference. And then I wrote an article when the 8T and 8i came out around that time, I’d been talking about like, hey, it’s the right, you need the right silicon for the right workload these days. And we’ve talked a lot about different memory hierarchy decisions and different networking decisions, not only with the TPU, but also with other AI accelerator startups that are like, oh, they’re choosing more SRAM here and more HBM here or also using DDR and all these different decisions. And so it made a lot of sense when Google came out with the TPU for training and the TPU for inference because the training one was for the more compute bound workload, and the inference one was for the memory bound and so it had more HBM. And then to your point, they had different networking and so you could see topologies, whether it’s the 3D Torus or the Boardfly. And so you could see, oh, hey, this is pretty cool. Google is designing not just chips, but entire system for training, an entire system for inference. So then when I saw that news, and I saw it just kind of fly past on my screen, and it was like, oh, also they have the 8TX and the 8i Max or whatever, I was just like, wait, we already have a system for training and a system for inference. What are these two other systems for? So, with that, what are they for? Let’s start with the 8i Max. Should we start there?

A Refresher on 8T and 8i

Austin: This episode of Semi Doped is presented by Crusoe, where you can fine tune and serve open models without managing a single GPU. When we sat down with Crusoe’s serverless fine tuning product team, we asked the question every AI user wonders, what will you do with my data? Crusoe has a simple answer. They don’t touch it or see it. Your data is yours to fine tune models with or simply store for future use. Their fine tuning is designed so that both your training data and the fine tune model itself remain your property. Your training data is encrypted, so even Crusoe’s cloud engineering team can’t see it. You can deploy it on Crusoe Cloud in one click or download the checkpoints and run it anywhere you’d like. The fine tuning process is fully managed, so you upload a data set, pick an open model, and Crusoe takes care of the GPUs, orchestration, and all the infrastructure management. You pay per token processed rather than per GPU hour, so run a few versions, compare them, and download the one that actually works best for your use case. Create a new Crusoe account for free and get $5 in credit to start testing today. This part is super important, guys. Support Semi Doped by going to semidoped.com/cruso, click on that link in the show notes, type it in yourself, and you’ll get directed to a sign-in page. Sign in with Google or create a new account, and Crusoe will immediately create that account for you with $5 in it. Boom. Again, just click that link in the show notes, sign in, $5 for you, and you support Semi Doped. So, thank you for your support. Go check out Crusoe, try to fine tune your own data. Again, the link to try serverless fine tuning is in the show notes. So, hit that now, click through, and make sure you claim that $5, give it a try, and support us. Back to the show.

Vik: I want to go and just set the stage for what the original chips were so that we could discuss what the new ones are because I want to just remind listeners what the previously announced ones this year were. So let’s start with the 8T. The 8T was basically for large scale pre-training, right? It’s a training chip. And the 8i was for inference, like that much is clear. The 8i had more HBM than the 8T. So it had 288 GB for the inference chip and 216 GB for the training chip. And the HBM bandwidth per chip was also higher in the inference chip. What was significantly higher in the inference chip was on-chip SRAM. It was almost three times as much. It’s 384 MB of SRAM on the inference chip versus the training chip. And you can see the chip too, it’s a humongous chip. The inference chip is humongous compared to the training chip and at Hot Chips the speaker from Google put up a picture and had a quiz and said, which one is the training chip? And everybody feels like they want to say that the training chip is a bigger chip for whatever reason you think the training is a hard problem and therefore bigger chip. No, no, it’s completely the opposite. It’s actually the inference chip because of the massive SRAM capacity in it was actually the inference chip. So the training chip was smaller.

So what Nvidia’s design choice for the inference chip was that they put in a lot of memory. So whether it’s HBM or SRAM, it’s memory. And this is a good reason for that is because inference is a memory bound problem and we will need this piece of information to discuss the 8i Max. That’s why I’m setting the stage. So we need to have a lot of memory because inference has basically two stages, the pre-fill and the decode, and the decode is an entirely memory problem. So it makes sense for an inference chip to have a lot of memory. So that’s why they did it that way. And the other way is that the training chip actually has more compute than the inference chip. So that’s the internals of the chip. There’s a little bit, like you said, there’s a system involved here really. So there’s a little bit more detail and that is for the scale up networking of the training chip, they use a 3D Torus topology. So this is what they have used for the longest time really with all their optical circuit switches. It’s a very complicated 3D interconnection that involves both copper and optics. And then the 8i actually uses Boardfly. And this is something to remember, and we did the whole video on Boardfly, but it’s important to remember that the inference chip used Boardfly. And we’ll see why this is important when we get to the 8TX. Because it’s very, very confusing. And then finally, both of these 8T and 8i as a head node, they use the Google Axion CPU, which is an Arm-based CPU.

Austin: Yes, yes, I’m with you so far. And yeah, I was just going to add in the 3D Torus, people can remember, I think of the Rubik’s Cube where it’s like, oh, I want to talk to my neighbors in different directions, and that makes a ton of sense for training because there’s very consistent communication patterns. And then the Boardfly is good on inference because these days we’ve got mixture of experts and you don’t know exactly who you’re going to talk to. It kind of depends on which expert’s getting fired for which token. So you do need a different topology that’s suited for that.

And then I’ll add in lastly, I’m glad you called out the Axion CPU and that it’s Arm-based, and it reminds me whenever I listen to Arm’s earnings calls, I listen to Intel’s earnings call and then AMD’s and then Arm’s, and in somewhat of that order, maybe Arm’s came first, they all run together. But, there’s always like an X86 one that’ll say, oh yeah, we sell, we still have the majority of CPU server sales or whatever. And then Arm comes in and they’re like, yeah, we sell, or 60% of the CPU server sales are Arm, and you’re always like, what? I’m so confused. These add up to more than 100%. But it’s usually like one is focused on just X86 and then Arm is focused on everything. And so head node, I bring this up because head node, because we tend to think like, oh yeah, Axion and it’s in Google Cloud, but you also forget they can count head nodes. So if you see, oh, a bazillion TPUs got made, then you know, I guess, well, there’s a bazillion head nodes to go with it or a half a bazillion or whatever. And that right now for this generation, it has mostly been their Arm-based Axion, but we will get into that as well in a little bit.

Vik: Yes, it’s a very, very important fact that Axion is going to be replaced by something else. But yeah, so this is the difference and the Boardfly topology was used for inference fundamentally over the 3D Torus as a refresher from the other video if you haven’t seen it, is that it reduces the overall number of hops required to go from one point to another within the networked GPUs or TPUs. And reducing the number of hops is important because you can reduce the latency. The less the networking you use, the better. So that’s the reason Boardfly is good for inference because we want latency to be low. So that’s where we are now. Now is a good point to jump into like you said, the 8i Max.

The 8i Max: Double the Memory

Vik: So the 8i Max is nothing different from the 8i. The silicon is exactly the same. They just took two 8i chips and put them together on a single package. That’s pretty much it, right?

Austin: Yes, yes. And there’s an amazing graphic that they made that we will try to include here that shows, yes, I think one server node or one pizza box if you will. I believe it used to have four individual chips. And now it has those same four columns if you will, but two chips kind of per column. So there’s eight in a pizza box instead of four. At least that’s what their graphic illustrated.

Vik: Yeah. So now the question is, why would somebody take the same chip and put two copies of it and then package it together?

Austin: Exactly. Or why didn’t they do that in the first place?

Vik: That was my thought. So there’s always an evolution because now what happens is normally people don’t want to jump generations like that because you want to make sure one chip works and before the other chip comes up because now if the package doesn’t work, two chips don’t work. By the way, I have to tell you this funny story. This is a true story from my personal experience in the semiconductor world. I worked at a company once that had to make two chips like this. I think it’s a great time to tell this story. And they decided that for whatever reason, because RF is very sensitive stuff, they decided they will make the mirror version of the same chip because the package required two chips. So they decided they’ll just mirror the chips so that you could create a symmetric package layout to put both the chips on. And they said, fine, this is a beautiful approach. Look at the symmetry and the beauty of this. RF guys love symmetry. And then what happened was the first iteration of the chip, something went wrong. Now the problem is the mirror image of the chip is not the same chip. They’re two different chips.

Austin: Oh, what? Oh yeah, yeah.

Vik: So for every iteration, yeah, because the complete the mask is a mirror mask. So even the transistors are mirrored. They are not like everything is not the same. So they had to run the same chip twice every time and pay for all the masks from beginning to end because they decided to use a mirror without determining if the first chip worked.

Austin: Oh, oh, no. So they, okay, so the first chip if it’s broken and then you create a mirror of it in an expensive way. It wasn’t just a rotation of the first chip. It’s an actual mirror. So you have all new masks and then, oh crap, the first one didn’t work. And so now our mirror probably doesn’t work.

Vik: No. It’s a mirror. So the development cost for this chip was double what it should have been because they mirrored the chip, and the chip didn’t work once. So every time you made a fix on one, you had to make the mirror chip fix and you had to double everything, mask costs and everything. It’s like, oh my God. So you know what? Make sure one chip works first before you make two of them. I have seen this in my own little world, and so Google obviously know what they’re doing. So, obviously the 8i is a good way to do this. But then the question is, why would you put two of them together? I think the advantage is that you double HBM, you double the SRAM. Remember it was 384 MB of SRAM per 8i chip. Now you have double that. You have 760 whatever MB of SRAM. Imagine what amount of stuff you can do with that. So now your bandwidth is enormous bandwidth, and your HBM capacity is so much more. It’s really good to hook up two chips. Everything in terms of hardware specs kind of doubles almost. And that’s it has its advantages.

Austin: Yeah, let me ask you a question about that then. I guess you’re making the assumption that the program has access to all the chips that are on the board and therefore to the programmer, suddenly there’s just twice as much SRAM and twice as much HBM. Is that how you’re thinking about it? Because if there are individual programs running on each chip, then it’s like, oh, well, they’re just in proximity to each other. So clearly these have to be like the system, the kernel can see now double the HBM, double the SRAM.

Vik: Yeah, I think that’s how it is because the way when you package things together in an advanced package or something like this, the idea is that they behave more like one chip working together, like it’s more like one huge chip.

Austin: Yeah. Like one monolithic chip. It just happened to be disaggregated but then reconnected through advanced packaging.

Vik: Yeah, when you use advanced packaging, the benefit of that fine grain nature of advanced packaging is that this 8i Max chip is literally like a chip that is two times bigger than the 8i. And you can’t make two times bigger chips because they’re already very reticle limited. Remember I was saying how big the 8i chip is to begin with, you can’t make it. You become reticle limited on the wafer anyway. So Nvidia does this too, with their Rubin packages, they have two Rubin die in a package and they were talking about putting four Rubin die in a package in an advanced package and things like that. So you’ll burn a lot more power per 8i Max, but you get so much more out of it. And this is good because this is all towards a trend of rack densification. How do you put more TPUs or GPUs into a rack? It’s very important to get as much compute as possible in a rack because networking to get the same kind of end result is a very difficult thing to do. And what I mean by that is you could just put two chips instead of advanced packaging them together. You can just put two separate chips and hook them up with even optics. Why do people not do that? Because there is no match to putting two chips in an advanced package. Optics can’t match that kind of latency. It’s always going to be slower.

Austin: And it costs more power.

Vik: Yes, exactly. If you’re moving bits over a longer distance, you’re going to burn more power. So nobody wants that. So you want to put everything into a single rack and densify the rack as much as possible because another benefit is that maybe you can do it with copper. The reach is limited. It’s the reach limitation means that you...

Austin: Yes, I was going to get there too. I had had the exact same thought, which was essentially, yes, we haven’t talked about moratoriums and stuff like this, but that’s where my head went. It was like, well, I guess if it’s going to be harder, which time will tell, if you’ve already got space and you can take one rack out and put another rack in and get twice as many, of course, that is a good thing for continuing to scale the AI build out for sure.

Vik: Exactly. So it’s a good thing. And the benefit is that the 8i Max now has so much more it can hold so much more of the model or it can hold so much more of the KV cache. And all that kind of stuff, you have to move data less, so you can save power. You can run long agentic tasks because you have so much storage lying around. You can run fast inferencing because you have double the SRAM. So it gives a lot of benefits in terms of what kind of performance you can extract out of a huge chip like 8i Max. So I think it’s definitely built for long context agentic workloads.

Austin: Yes. So I guess, okay, two thoughts. One, I remember Nvidia was going to go to tying four reticle limited GPUs together, but then there was a question about warpage from CoWoS because it means you needed a bigger substrate. And then it made me think, oh, is this is just going from one to two, but I’m assuming that the TPU is also built using CoWoS. I don’t know if it’s CoWoS-S or CoWoS-L. And so then I’m thinking, what’s the TPU going to do in the ninth generation? Do you think that they will still have the one and two variants, like the 9i and 9i Max, or you think they’ll just go with this Max? Because don’t we always just want more memory for inference? And then of course, then I also wondered, are they going to scale it even bigger to four? Or is 9i going to have two and then 9i Max actually has four? What do you think?

Vik: Yeah, I think they should go the 9i should be whatever the 8i Max is. That’s the logical way to go. And I don’t think they’ll... I mean, the 9i Max might be the four version. Total guess, by the way. I have no idea what the road map is. I have no inside information or any such thing. I’m just totally guessing here. 9i Max may have to go and solve those warpage issues and then we can put more chips. And I’m guessing Nvidia will also reach that point where they will put four Rubin die into a... I think Rubin Ultra, right? That was supposed to be for the Rubin Ultra. And so they may reach that point too where we can make this big enough. All these advanced packaging houses are working towards making bigger and bigger packages. We’re talking about glass interposers and so many things are coming down the line. Those are all going to be solved eventually and so we will go to four-die systems. But yeah, that’s the way to push the technology forward.

Austin: Yes, come on, Intel, EMIB, get in the game here, work with Marvell and whoever, MediaTek, make these TPUs, try to package them. Oh, and I’ll also say the Max naming reminds me of the iPhone, and it feels like we’re talking about iPhone generations because it’s like, oh yeah, this year’s Max will be next year’s base model.

Vik: I think they’re losing the naming game. The 8T and 8i was nice. But now it’s going to get out of hand. Do we have to cover anything else about the 8i Max or do we go to the other crazy name?

Austin: I don’t think so. I think it makes sense. Let’s go to 8TX. Okay, 8TX. What does that mean, Vik?

The 8TX: A Secure System

Vik: Okay, so what is the 8TX? So let’s get into that. It’s a weird name and I have no idea what the name means, but Austin, you will suggest a better name at the end, I’m sure. Because you have a flair for these kinds of things, but I’ll just explain what it is. So it takes the 8T silicon, which is basically the smaller chip. Remember, this is the one with less SRAM, less HBM. And then throws out the Google Axion host node CPU and instead puts a fifth gen AMD EPYC CPU as the host node CPU. So it uses the 8T, the training version of the silicon, and it uses the AMD chip instead of the Arm-based Axion chip. And funnily, it uses the Boardfly topology. So remember the 8T silicon that we spoke about used 3D Torus because it was a training chip. But this uses the 8T silicon with Boardfly topology and an AMD CPU. You see the confusion here, it’s a complete rejigging of everything.

Austin: Right, right. It’s like the training chip with the inference network and then with a different CPU. I would not name it. I would not put a T in it. Now, well, I guess, okay, so T is use case to me, it’s training, right? So what did they say the use cases are for this, by the way?

Vik: The use case seems to be what they call confidential inference and we’ll get to it, but that comes from the use of the AMD EPYC CPU. And they also said this used for private fine tuning of models.

Austin: Okay. So I think the first bit was inference and then the second bit is training. But security sounds like the most important thing here. So maybe the best name would be 8S.

Vik: Yes, good. Let’s go with that.

Austin: Because then it doesn’t conflict the use case and the topology and stuff. So maybe, so Google, if you’re listening, maybe next time chat with us, maybe something like 8S, eight secure chip.

Vik: Eight secure is so much nicer.

Austin: System, system. It’s a system.

Vik: Yes, it’s a secure system. See, you just have to use S. And 8TX, I don’t even know what the X stands for.

Austin: Well, I mean, for an RF guy, TX probably means transmission.

Vik: That’s what I thought. Yeah. But it’s not that. Okay, so confidential... so what is the use case for this? I think that essentially this chip is designed to make sure that you can use these chips to do AI in information sensitive environments. Like a trading firm might be training on their data to make stock purchase decisions. It’s very important for their business and it involves a lot of money that they’re training towards making maybe these trading decisions, maybe hundreds of millions, billions, requires security. Nobody can access this. I think of somebody like Jane Street or a big company like that doing something like quantitative trading, high frequency trading, things like that. Or the other use case I can think of is think about Nvidia, for example, or probably a bad example, but anyway, think of any company that has a super secret way to design a chip. This is their secret sauce and they want to now train AI on their secret sauce chip design. And they don’t want to run it on a server that’s shared by anyone else. The thing is, if you go to the cloud and you start running inference, your hardware is being shared with other people who are also running their inference on yours. So how do you guarantee that your tokens or your information or whatever is not going to mix in with somebody else’s or leak in any way? There’s no way to guarantee that. So I think this chip is based towards that.

Austin: Correct. Yes. I think that is actually the key and primary use case, which is Neoclouds or clouds or whatever and having neighbors potentially sharing the same system that you don’t want to see their information. Because prior to this, if the first and best customer was Google, well, of course, when they’re training Gemini and all this stuff, wouldn’t they want it to be secure? Why didn’t they build this security in hardware security in mind in the first place? Well, yeah, of course they want to be secure, but they were the only customer. It was all just internally used compute. So now when they’re trying to give their compute to someone else who’s going to be a landlord and rent it out to other people, I think now is where you want to say, oh, yes, you can put your data on our accelerators and when it moves from the accelerator to HBM or from the accelerator to the CPU or whatever, it will move encrypted data is my understanding.

Vik: Correct. So, the main purpose of this is to make sure that the data only reaches the target it is intended to reach. And by that, I mean the data is encrypted. And so there is a key. The key is held only by the correct party who can unlock the data. So, even if this data goes to some other place by whatever means, they can’t do anything with the data. There’s no key to unlock it. So, that’s what this whole confidential computing comes to. So, only the intended recipient can interpret that data and that kind of encryption is very important for trusted, multi-tenant situations.

Austin: Yes. Yes. Absolutely. Totally. That makes sense. Okay, so why an AMD CPU then? Why not the Axion?

Why AMD? And Why Boardfly?

Vik: So, Axion does not have the feature to basically encrypt this information to the level at least that AMD can do it. I don’t know if that feature exists or not, maybe it does. But I don’t think that it’s as good as what AMD can do with their EPYC systems.

Austin: Right, right.

Vik: So, the AMD chip has something called a secure encrypted virtualization or SEV. So, essentially, you can have a virtual machine that is completely isolated from other virtual machines and from the underlying operating system that the virtual machine sits on. That underlying operating system is often called a hypervisor. So, the VM is completely insulated from the hypervisor underneath. Hypervisors are shared by many VMs. So, you don’t want any information leaking from VM through the hypervisor to the other VM. So, this capability allows not just any kind of VMs hosted on these CPUs to be secure and encrypted, but also it allows information that is coming in and out of the CPU or devices that are connected to it, like the TPU in this case, the 8T training TPU. All of them inherit this secure encrypted virtualization feature, which is what allows secure transmission of data between these various parts of the system.

Austin: Gotcha. So, the 8TX is using the 8T silicon and nothing has changed. So, all of this is done in software, managed by the CPU, but the TPU must have some sort of software to be able to handle this as well.

Vik: I’m not sure of that software layer within the CPU, but there’s nothing new in the hardware because the silicon is exactly the same as what came up earlier. So, the CPU orchestrates the confidential computing. That’s the key thing to take away here. So, all that encryption process, all that is handled by the CPU. And that’s why they had to go and use this Arm CPU instead. And actually what is funny is that during the announcement, AMD actually put out a press release as well talking about their SEV feature, which I don’t know, never made it into the news, but if you see it’s there. They announced that it’s there. This SEV is why it is inside the Google 8TX.

Austin: So, yes, the natural question by the way to that is, okay, is this just a temporary lead? Like AMD got it because they have the SEV and will Arm end up saying, okay, that’s on our road map now. So that the next Google Axion can have it or is this like a sustainable advantage for AMD?

Vik: No, no. So, I think that one of the reasons is that Arm does have client confidential computing. I think they call it client confidential computing architecture or something. If you see their Arm AGI, this phrase is in there. So, Arm already has a CPU that can do this. I don’t think it has that kind of flight history that AMD has, because AMD has had this feature for a while. So, I think it’s not the right time. I don’t think it locks somebody out forever. Other people have it too. Intel has a version of this as well. And I wrote it down here and it is actually called trusted domain extension or TDX. And I was looking up like why did Google actually choose AMD over Intel for this chip. Why don’t you get an Intel chip? I just saw that there was some security concerns and stuff with the trusted domain extension. So, I figured maybe they just didn’t want to deal with it and they went to the AMD one because it’s well established and nice to go.

Austin: Or yes, yes. Or maybe they weren’t sure there would be enough supply from Intel. Hope that’s not true. But you know, maybe they just said, hey, AMD seems to have the supply stuff figured out and we’re going to need a bunch of these. Now, yes, to your point, I agree. Clearly, the Axion was made at a specific point in time with certain design decisions that made sense for the Google Cloud. And so in this case, when they’re like, hey, we need this security stuff, why not just pick one up off the shelf from a merchant silicon vendor who already has a portfolio of CPUs. So, obviously, AMD already had silicon that can do this, so does Intel. Maybe in the future, we’ll see a portfolio of Axion, you know, this is our head node one for the secure environment and stuff like that. But point in time, easiest to just pick something off the shelf, plop it in.

Vik: Yeah, what is so funny about this announcement is that at this time, I got an X invitation. Somebody followed me. I’m like, oh, who is this person who followed me? Oh, it’s Arm. Oh wait, Arm is following me. That’s cool. And then I’m like, oh wait, Arm DM’d me. Like what did they say? They’re like, oh wait, Arm DM’d me to say that the 8i Max actually uses the Arm-based Google Axion. Because it wasn’t clear from the press release what that was using. The 8TX uses the AMD. Yes, we know. But what was the 8i Max using? So, they did it to a bunch of people on X.

Austin: Okay, so they were just proactively telling people on X. Had you been... okay, okay. I was like, did you talk about it and then they DM’d you, but no, they’re just proactively.

Vik: They’re proactively telling people on X.

Austin: Good job, Arm’s comms social media team for proactively saying like, just in case it wasn’t clear, we’re also in this slot here.

Vik: Yes, please send it out on your channels just so that we know. So, I mean, I’m doing my part. Semi Doped is where I talk about this stuff. And yeah, so 8i Max uses the Google Axion, like everything else. And the same reason is simple. There’s no encryption-based confidentiality computing here. It’s only a 8TX feature. Can we just call it 8S? So, that’s the whole point. So, that’s where we are right now with these chips. I think it’s a good addition. I think it unlocks... the 8i Max is honestly a bit more exciting to me because of the sheer size of that chip.

Austin: Yes, but you can fine tune on it because it is just an accelerator at the end of the day and like a GPU, you can run several workloads. And I think the interesting point too is that really one way to think about the 3D Torus is it’s also very good for large scale training when you’re like pod level training, you know, like I need to talk to the chips that are closest to me on this board and then the chips that are closest to me in this rack, but then I also need to be able to talk to maybe other pods, using RDMA. And of course, yes, Boardfly is more about minimizing the hops because you don’t know who you’re going to talk to yet, you don’t know who you’re going to talk to next, but if you’re doing small fine tuning, you’ve got an open like a some medium-sized open model and you’re just taking your own internal data that you want to be secure, and you are doing some training. It will still work with Boardfly. It’s not like it won’t work without the 3D Torus. It might not be the most efficient thing in the world, but at the end of the day, you’re not the kind of person that’s like, oh, I’m putting 100,000 TPUs together and if this takes me an extra day, it costs an extra couple million bucks. No, you’re just fine tuning small stuff that probably lives all within one rack or maybe even within one server node or pizza box, so to speak. So it can it’s inference first, but it can be used for fine tuning.

TPU Sync: Google’s Software Layer

Vik: Yeah, yeah. Part of this slew of blog posts that came on the Google Cloud blog, I mean, if anybody wants to look it up, it’s very interesting. They had so many other blog posts that was relatively lost in the noise. One of them was called the Google Kubernetes engine. It’s called the GKE and it does exactly what you’re talking about. It’s called dynamic slicing. And the whole point is very nice actually. And there are really good animations. We’ll leave links in the show notes. But essentially, you can connect these groups of TPUs together to form a bigger pod. You can take many smaller pods of GPUs and you can hook up to them to become what they call a super slice. And then you hook up all the external connections via optical circuit switches, OCS. So you can connect smaller pods into huge big pods with OCS and that’s called super slicing. But you can do the reverse too. You can go into a single pod and just carve out a few TPUs. So basically, you have complete flexibility, you just throw a bunch of TPUs into a data center. You can slice them any way you want. You want to put them together to make a bigger thing, you can do it. You want to take a single one and break into it to sub slicing, you can do it. It’s fascinating the kind of things that you can do with these things.

But yeah, all that is really detailed. The blog posts as usual from Google have beautiful animations and explanations. They’re not even very long, so I highly recommend people read the links in the description. But I think one other point we should touch on now that you mentioned that other people like Oracle can go and use Google and they have some announced some plans around this too, is that what makes this happen? How does Oracle use Google TPUs without... CUDA was the biggest thing for Nvidia, like everybody could program on CUDA and use GPUs. I think Google is making it a lot more easier now to do those kinds of things with TPUs. And I’m no software expert, so I can’t go into all the details, but I think one thing stood out to me nicely that straddles the world between software and hardware and that’s called TPU sync. So TPU sync is now an open source project that essentially, I feel like it’s Google’s version of Nvidia’s Dynamo. Maybe I’m wrong, but somebody can correct us. But basically what it helps you do is that it helps you manage the memory hierarchy. It allows you to handle HBM, DRAM, SSD storage, and even network storage. I think Google has a special version of network storage that they use. So all of that, this software layer, which is open source, allows the inference engine to really distribute work through the right places at the right times. It doesn’t recompute anything. There is always a place in the memory hierarchy to store something. And so that’s what TPU sync is. And we’ll get into a little bit more detail, but what do you think of this software layer?

Austin: Yeah, yeah. No, I think it’s very important. Obviously, if you’re trying to sell your Silicon, as AMD knows, if you want to sell accelerators, but people aren’t happy with your software, you’re not going to sell accelerators. That was the ROCm thing that they had to deal with like two years ago. They’re pretty much past it now. No one talks about ROCm anymore because it just works. So clearly, if you want to use TPUs, you want to sell them to external customers, the software has to be very easy to use. And to your point, a big part of LLM workloads at scale on accelerators today is disaggregation and then it’s also memory management. And actually, I think this probably fits in nicely too with having the 8i Max is like, oh, some of the chips have a ton of HBM and a ton of SRAM and how can you make sure that you put the hot KV cache stuff where you need it and offload some of the colder stuff to still have it something that you can access, but not as close. And it actually it reminds me of the conversation you had with Val Berkovici of Weka. You guys talked a lot about this like LLMs needing to think very carefully about having as much essentially having a memory hierarchy and can you also do tricks in software to make it feel like some of this stuff like flash or something is lower latency than you might expect because you’re thoughtful about what you put where and how you fetch it in time.

Vik: Yeah, exactly. It feels like TPU sync helps with that. It helps with that. And the blog post I was reading says that the amount of the KV cache transfer overhead drops by like 90%. There are some time to first token improvements that span anywhere from 10% improvement all the way to 90% improvement depending on various configurations and model sizes and things like that. And also the time between inter token latency, it’s called ITL. It’s just the time that is taken between tokens, that comes down as well with the use of TPU sync. And there’s a nice analogy in the article, which is basically like think about a very fast executive chef at a restaurant. Now, the guy is cutting all the vegetables and cutting all the meats and cooking and but he has a very small countertop. So he can’t literally keep all the meat and vegetables and the food or whatever that he’s cut. So what he has to do is he has to cook it and then when the next batch comes, he has to cut everything from scratch and store it in the small countertop. He needs to keep doing this. It’s very slow. So the TPU sync idea is that now you don’t have this small cutting board where he is operating. Now you roll up a cart next to him where he can store all the food that he’s cut. And if there’s meats, then you have a cold storage there where you can store cut meats even if you wanted to. So everything is in its right place when you need it. So the chef only needs to get stuff from the right place. Sometimes, yes, it takes time to go to cold storage to get the meat, but it’s far better than cutting the meat from scratch.

Austin: I love the analogy. And it just goes to show, when you’re operating these workloads at such big scale, it’s very important to have the right stuff at the right place at the right time. And co-design and co-optimization from hardware into software can make this possible. So, what I just thought when you were explaining the analogy even, it just made me think like, oh, what a cool job for some people out there, hardware and software engineers working on this to think about like, oh, how do we make sure that the KV cache is in the right place at the right time and how can we write software to make that even easier? Because obviously, it feels very technical and kind of like a niche problem, but even like inter-token latency, that kind of stuff could impact millions or billions of users, like they they would legitimately feel that. Of course, that’s a whole another conversation of if Google can get people to use Gemini and, where’s Google’s agent and stuff like that. We won’t go there. But very cool, very cool technical problems to work on.

The Business of Selling TPUs

Austin: Today’s sponsor is Meter. You’ve got the GPUs and the power. Now, who builds the network and who do you call when it breaks? With Meter, it’s one call. Tell them your GPU or server count and they design, deploy, and run your front-end storage and management networks. Meter builds its own networking hardware, switches and all, and writes its own network OS. So, that call goes to the team who built it. Running a Neocloud or clusters in a colo, go to semidoped.com/meter. Links in the show notes.

Vik: Yeah, I’ll just explain maybe two use cases that are very interesting to me. One is, think about enterprise use cases. Before a prompt is sent by any employee, companies say they want to put a lot of guard rails and rules and everything into place. So, what they usually do is they prepend the entire prompt that you send with all the guard rails first. So, it’s like you send an instruction manual of what you can and cannot do before Austin can send a request.

Austin: Yes. It’s like attached to every email that I send.

Vik: Yes, yes, that’s attached to every email that you send. So, this kind of stuff can be stored with TPU sync in the right locations. So that you don’t have to recompute or redo this. You have the instructions already always and you can always process requests quickly. That’s one use case.

The other use case is that I think one way to drop inference cost is to make sure that KV cache is not recomputed. You want to have KV cache hits as much as possible. You want to use tokens that are already existing from the past or things like that. So, if you have all this storage hierarchy, you will have a much higher KV cache hit rate, which means you will drop your inference costs by an order of magnitude in this environment. So, it’s a nice, nice thing, I think. They also have a nice thing called host-to-host streaming, which means if you’re doing prefill on a separate rack, and you’re doing decode on a separate rack, they have these high-speed interconnect, it’s not interconnect, it’s like a protocol that they call host-to-host streaming, H2H, that makes sure that the entire weights from the prefill goes to decode quickly. So this kind of stuff, they’ve thought through every part of the memory hierarchy nicely. But I feel like this is not unique to Google. I feel like Google, Nvidia has thought about a lot of this as well. So, but it’s good to see the progress, right?

Austin: Totally. Totally. Agree, agree, agree. Yeah, and I’m sure AMD, for example, thinks about this too, but good to see Google doing it and to know, if you’re asking yourself, can they compete in selling these to people? Clearly this shows that yes, they can build this, people can be confident that they can run on top of this. And obviously, I would like to think that Google’s using this all internally too, so, you know, dogfooding it, so to speak. Which would be a different, and that actually could be an unfair advantage that they have compared to a merchant silicon vendor like an AMD or Nvidia where AMD and Nvidia don’t necessarily, yes, they do their own training, this could be a reason, a good reason why AMD bought World Labs, pushing into the having your own internal team to really understand the 3D world model type use case. But Nvidia obviously with Nemotron, they train their own model and it’s not because they want to undercut their buyers and say, yes, we’re going to try to beat Anthropic and beat OpenAI. No, no, no. They train their own model so that they can make these intelligent decisions about how to architect their system. But just saying, if Google’s not so big and hierarchical that they don’t talk, but hopefully their the Gemini team, the DeepMind team, etcetera, is talking to the TPU team a lot and so that they’re using this software and they’re talking and sharing all that knowledge as they’re building the systems that they’re consuming this all right away too.

Yeah, it’s exciting when people build exciting hardware, we are excited to talk about it. So, if you’re listening to this podcast and work in one of these companies doing all this cool stuff, you know, keep at it.

Vik: Indeed.

Austin: Indeed. Yeah, let us come by. Show it to us sometime. Have Semi Doped, have a Semi Doped tour sometime.

Vik: Absolutely.

Austin: Cool. All right. Well, Vik, anything else on the TPU front?

Vik: No, that’s it for now. We’ll leave all the blog articles on the show notes, but yeah, you can also go just find them. They’re really nice. I love reading Google Cloud blog articles because they have like the best pictures and the neatest explanations and they’re kind of short. So, it’s very much fun to read it. I always have fun.

Austin: Awesome. Yes, I love those too. Good job, whoever makes them. All right, with that, everyone, thank you for listening. Send this to your friends. We appreciate recommendations and like, comment, subscribe, whatever. Vik and I read it all, so please, you know, send us your thoughts.

Discussion about this video

User's avatar

Ready for more?