Vik Sekar talks with Sudeep Bhoja, co-founder and CTO of d-Matrix, about the company’s journey from analog in-memory compute to its current digital approach. Sudeep explains the architecture of the Corsair and Raptor chips, detailing how 3D stacked DRAM achieves SRAM-like bandwidth with much greater capacity and power efficiency. They also discuss how d-Matrix integrates into the Nvidia ecosystem to build multi-rack systems for massive AI models.
Presented by Crusoe
Fine-tune and serve open models without managing a single GPU.
When we sat down with Crusoe’s product team, Austin asked: what will you do with my data? Their answer: They don’t touch or see it.
Your training data is encrypted so even Crusoe’s own admins can’t see it, and the fine-tuned model is yours. Deploy on Crusoe in one click, or download and run it anywhere.
Upload a dataset, pick an open model, and pay per token rather than per GPU hour. Run a few versions, compare them, and download the one that works best.
Accounts are free, and get $5 in credit when you sign up.
Things we cover:
From analog to digital in-memory compute
The Corsair PCIe card’s SRAM and LPDDR
Raptor’s 3D stacked DRAM architecture
Speculative decoding and workload disaggregation
Building multi-rack systems with Nvidia’s Vera Rubin
The future Lightning architecture with four-high DRAM stacks
This episode is supported by Meter.
You have the GPUs and the power. Now who builds the network, and who do you call when it breaks? Meter designs, deploys, and runs frontend, storage, and management networks on its own switches and its own network OS. One call reaches the team that designed the hardware and wrote the software.
This podcast is lightly edited for clarity.
The Inference-First Approach
Welcome to the Semi Doped podcast where we have conversations with some of the best minds in the semiconductor industry. I’m Vik Sekar and I’ve worked in the semiconductor industry for like 20 years, and today I’m an independent researcher and the founder of Semi Exponent. I also write a substack newsletter called Vic’s newsletter. Welcome to the show, Sudeep. How are you?
Sudeep: Good, good. It’s a pleasure to be on your show, Vic. I’ve always been a fan of all your work, and it’s great to be on your podcast.
Thank you for coming on. We have so much to chat about today because I am quite familiar with d-Matrix. I’ve covered it on the substack before and we’ve spoken at conferences. There’s so much exciting technology to get into, so we should just get right into it. Let’s start right at the beginning because the story here is very interesting and that’s because d-Matrix actually was founded in 2019 and that was well before the AI thing even took off. You and your CEO, Sid, were both at Infi before starting d-Matrix. And it’s really interesting because right after that, actually Marvell acquired Infi. So that was really cool to see how you guys decided that you’re going to go and do inference architecture that early in the boom when something as exciting as Marvell’s acquisition of Infi was in the works. So how did you foresee all of this happening?
Sudeep: Of course, we did not know the acquisition was in the works at the time. But we were working at Marvell in the data center business unit and the interconnect products we were building were being used by the hyperscalers—Google, Microsoft, Facebook at that time, Amazon. And we were seeing that the AI training clusters were pulling our interconnect solutions. And they were in fact driving speeds and feeds, which was a change from what it used to be before that. And then we realized that they were training these models, these amazing models. ImageNet had come out in 2012 and there was breakthroughs after breakthroughs already. And late 2017, the transformer paper had been released.
We decided to focus on inference because we realized that the problems of inference were very, very different from what everyone else was working on at that time, which was training models. Training models is about high batches, compute, right? Because it’s a compute-bound problem. You want to pack as much compute as possible. Whereas inference is a memory-bound problem. Latency is very critical. And so, that’s what we focused on.
That is a unique decision to make at that time because if you actually see all the other, I’d say pre-ChatGPT companies, whether it’s Groq or Cerebras, all of them went for the training problem first because all of them wanted to go and train these models. And I don’t think at that time the industry realized how big these models are going to get and how many chips are going to be required. So a lot of them did start with the SRAM-based approach, which actually d-Matrix also has in their technology and we’ll get to that. But clearly you went straight for inference. So that is a unique problem that you identified early on for sure.
Sudeep: Yes. Specifically, we recognized that once you train these models, you have to run inference on them. And the characteristics of inference was very different from training. To do an inference, you have to read all of the weights for every token and read all of the user’s KV cache. So if you want to keep latency small, then you can’t batch too many users. And that means that the memory subsystem is very pressured. There is not sufficient memory bandwidth to feed any sort of compute. And having worked in the semiconductor industry for many years, we recognized that the compute itself was very cheap. It was in the femtojoules per operation. The data to feed the compute that had to come from HBM or other memory subsystems was in the picojoules. So there was this three orders of magnitude gap, which is the technical problem that we wanted to focus on and solve. And that turned out to be a very good match for inference.
From Analog to Digital In-Memory Compute
So that power delta that you identified told you that this is the problem you need to solve. So who was the first person you convinced that this problem is worth solving? Who decided to fund the effort first?
Sudeep: The initial check was written by Infi, the company we worked at before we started d-Matrix. So they were the first investor in d-Matrix in the seed round. And then on series A, we had Microsoft M12 and Playground Global as our lead investors.
That’s fascinating. It’s fascinating you could get Infi as your first investors. It’s amazing. But your first chip looked nothing like what your technology looks today, right? So what was the first kind of chip that you went after as soon as d-Matrix was founded?
Sudeep: We wanted to build a proof-of-concept silicon before we scaled the company in our seed round. So we essentially went after trying to co-locate compute and memory together. After many years of transistor scaling, we knew that the compute is very small, but the wires had not scaled at the same rate. So it was data movement that was a problem. Our initial idea was, can we take an SRAM array and add compute into the SRAM array itself? So unlike a traditional SRAM where we only activate one word line at a time, we ended up designing an array where we activated all of the word lines. That was a pretty good operation for a matrix calculation, matrix-vector specific calculation. And then we activated, instead of the bit line, we used an analog scheme there using capacitors to do the accumulation of the bit line.
Okay, awesome. So, just for some background, SRAM generally is arranged, let’s say in a grid, and you can activate it with word lines and bit lines. You can store something in an SRAM cell by applying the right voltages at these levels and reading out with the right sense amplifiers on either side and you can get what bit was stored or what was not stored. That’s typically how SRAM is used to store information. It’s been around a very long time. Typically there are six transistors that build up this unit cell which you access with bit lines and word lines. That’s the basic SRAM approach. But you were not using SRAM as a storage unit as much as you were trying to compute with it, right? You were trying to do analog compute using this six-transistor SRAM cell.
Sudeep: Yes. The weights were being stored on the SRAM cell. And then we added a few transistors to convert that into a multiplying bit cell. So if you insert on the word line, if you were to send activations one bit at a time, then you could essentially multiply right at the place where the data is stored. So there’s no movement of data. And then instead of the bit line, we ended up putting capacitors on top of the SRAM array and shorting the bit line. And the capacitors would essentially do the addition.
Awesome. So at the heart of all of this is the multiply and add function, the MAC operation that you’re doing, but you’re doing it now in the initial version of the d-Matrix, the first version, it was all analog. And it was done by holding the weights in the SRAM cell and using an extra transistor to do the multiplication and then a capacitor to do the charge addition of all of this. And then the result you read it out and then you have your MAC operation, which means now you can multiply vectors together.
Sudeep: That’s correct. That’s exactly correct. In the original versions, we have digitized a lot of it. Obviously, the storage is binary, so it’s a bit-serial, bit-parallel architecture. We have removed the DAC function that traditional in-memory compute circuits had by sending one bit at a time. And only the accumulation was analog. That was the first test chip that we built, the proof-of-concept chip. But that proof-of-concept chip was a full quad, if you will, of the Corsair product itself. It had a RISC-V processor, it had all of the architecture built around these compute engines. It had a software stack, it had a runtime. So we were able to prove out the whole stack using these basic fundamental building block elements that did matrix-vector multiply, SIMD operations, reshapes, data movement, and so on.
What was the first version of this chip called?
Sudeep: It was called Nighthawk. That was the chip that we taped out in our seed stage of the company with very few people. And then we realized along the way that it was very difficult to maintain precision of the bit line. Customers are very used to seeing 32 bits of precision. And so we had to maintain that level of fidelity. And occasionally we would see a bit error that we worked very hard to correct, but it was very hard to maintain 32 bits of precision across parts when you scale this product. And that’s the reason that we went down later on on Corsair, we went down the digital in-memory compute approach. What we did is we just took that analog bit line and digitized that bit line using an adder tree. And we had worked on low-voltage circuits and we looked at the energy consumption of that versus analog and then we decided that it was far easier to productize the digital in-memory compute.
The Corsair Architecture
Okay, that’s a perfect segue to go into the next product you have because clearly when customers wanted that level of precision, you can’t do it with analog compute. Today, the fundamental method of computing you have is digital, right? So now you have digital in-memory compute. Can you walk us through what the transition is now so that you have it all digital? How that looks?
Sudeep: The matrix multiply units are exactly the same, which is you have the SRAM array and as part of Corsair, we have all of the word lines that we activate and we perform this bit-serial, bit-parallel multiply and add circuitry in the tensor engine piece of it. And instead of an analog accumulation, we put an adder tree and that’s how we are able to perform matrix multiply in digital and we get to very, very good energy efficiency by doing that.
I see. Okay, that’s good. So now your entire compute is still digital in-memory compute, but now you use entirely digital techniques, so you get the kind of accuracy that people were looking for.
Sudeep: Correct. We got to 32 bits of accuracy, so we maintain FP32 precision on the accumulation data path. So it’s just like any other digital engine from that perspective. It’s just that the computation happened right at the bit cell. So there’s no moving of data inside the tensor engine. And for that reason, those tensor engines are very efficient. They get to 25 to 40 TOPS per watt, depending on the voltage and the frequency we choose. Some of this we presented at Hot Chips last year.
Nice. I had one question though, because if you have FP32 and that was probably the initial implementation of what people thought inference should look like. But today we have other formats. We can do FP16, FP8, FP4. Do we still need that level of accuracy when it comes to these operations? Or could you get away with something far less accurate?
Sudeep: Even on Corsair, the operands we supported block floating point format. So we supported something called MX int8 and MX int4. So those are like 8-bit and 4-bit formats that shared an exponent across a whole vector. Now it’s part of an OCP standard. So the precision we support is 8 bits and 4 bits. But only the accumulation. The activations could be in 8 bits, the weights are in 8 bits, but when you add a lot of them up, the accumulation is now in 32 bits. That’s what we used. So we certainly don’t need 32 bits for weights and activations.
I see. Okay, that’s a good distinction to make. The thing with this digital approach, which is now your Corsair platform, is still completely SRAM-based, right? And so what that fundamentally puts you in is now in the class of the other SRAM-based accelerators, which all have a fundamental limitation and that is capacity. So how much capacity does Corsair have and what is the implication of that limitation?
Sudeep: Here’s Corsair. This is a PCIe card and we have two gigabytes of SRAM on this card.
So that Corsair PCIe card you’re holding up essentially has not just one chip in it, right? Like how many chips does it actually have?
Sudeep: We have, if you take all the sheet metal out, this is what the card looks like. So there are two chips. Here we have taken out the lid of the other chip and you can see the four chiplets on the second chip. Each of the chiplets is half-reticle. So there’s a sum total of two reticles on each chip. And together they form two gigabytes of SRAM capacity. But more importantly, they can deliver 150 terabytes per second of memory bandwidth, which is orders of magnitude more than what is available using HBM. Even Blackwell GPUs are 18 terabytes per second and this is 150. So this can deliver a lot more bandwidth because it’s all on-chip. And at the back of the card, you can see all of the LPDDRs. So that’s what we use to store as bulk storage.
Is it the black chips that are the LPDDR in that image?
Sudeep: Correct. Black chips are the LPDDRs.
Okay, so that’s interesting that you have both SRAM and LPDDR and we’ll get to what the benefit of that is. If you have two reticles worth, I’m guessing this is a TSMC leading-edge process. What node is it built on? The Corsair.
Sudeep: TSMC 6 nanometer.
6 nanometers. Okay. That’s a good node in a sense SRAM kind of has stopped scaling much beyond that. So if you go down to like maybe two nanometers or three nanometers, you don’t get as much SRAM capacity anyway and then you get the benefits of better wafer cost at 6 nanometers and therefore you still get 2GB SRAM out of this.
Sudeep: That’s exactly correct. The SRAM roadmap like you pointed out has stopped scaling. You only see maybe 10% better density as you keep shrinking process nodes from 6 nanometers to 4 nanometers, three and two and so on. So 6 nanometer is a very sweet spot in terms of SRAM capacity.
Disaggregation and Use Cases
It’s useful to put what this PCIe card is capable of because it has only 2GB of SRAM. Clearly to run a model that’s bigger than that, you’re going to have to put more of these cards together, right?
Sudeep: Yes. Exactly. We use it for many different applications. We build up whole racks. We take a server with eight of these cards, for example. We have a Supermicro X14 server with eight of these cards. And we also have low-latency networking that we built called Jetstream. And we use that to build up an entire rack that we call Squadrack with an Arista top-of-the-rack switch or any Ethernet top-of-the-rack switch that can host the entire model, all in SRAM. So per rack, we can get 128 gigabytes of SRAM.
That’s pretty good. So for sure you can run probably not like the biggest models out there at 128GB, but you can run some quite capable models at blazing fast speeds in a single rack.
Sudeep: Correct. Exactly right. There are different parts of the workload that we disaggregate and run on our cards. So we think of our deployments as heterogeneous deployments. There’s still a GPU. And we accelerate the GPU, we turbo-boost the GPU. And for example, one use case is something called speculative decoding where we have a draft model that guesses what the target model running on the GPU is and helps accelerate the GPU’s target. For example, the GPU could be running Kimi K3 or GLM 5.2, a very large model. But since the GPU is running at one token at a time, it’ll be relatively slow. But if a drafter that’s running on Corsair on the SRAM can guess, let’s say eight or 16 tokens at a time and provide that guess to the GPU, the GPU can verify that in a single shot. So that’s the speculative decoding use case that we have talked about.
The other application is something called AFD, attention feed-forward disaggregation, where we take the experts—the experts are very memory-bound—and we move the experts to Corsair, to the SRAM chips, but we leave the attention and the pre-fill on the GPU. So that’s a second use case. And the third use case is you leave the pre-fill on the GPU and only have Corsair cards to decode. So there are different applications, each one disaggregating a portion of the workload that works well with these fast-decode, limited-capacity chips.
Okay, that’s a great description because what the framing that comes out of all this is that you have to use this chip where its strengths lie and its strength lies in providing an enormous amount of memory bandwidth, 150 terabytes per second, which is like 10 times higher than what GPU HBM-based systems give you. And I believe there is also a benefit that comes from not having to move bits as much and that saves a lot of energy as well, if you don’t have to shuttle bits far away, right? So we’ll get to the energy efficiency aspect. But because of this capability of Corsair and SRAM and the limited capacity, you can actually use it in unique places.
The speculative decoding was one example you gave where, I’m just trying to explain to myself so I understand it as well, is that you use a smaller model. The drafter model you mentioned is usually a much smaller model than like a Kimi K2 big, trillion parameter. This is not a big model. If my understanding is the speculative decoding drafter model doesn’t even have to be that accurate. So it’s okay to use a small model because you’re going to be spraying out like eight tokens and the GPU will decide which of these eight tokens are the correct one by the process of verification. So the idea is that you have to keep the GPU fed with speculative decoder and the drafter, which is being really fast, which is where the Corsair fits in. So that the GPU can run in parallel verifications. It can run 8 to 16 tokens in parallel verification as long as it gets enough tokens from SRAM to verify, right?
Sudeep: Exactly correct. Some of the work we did with Gimlet, we have a 1.6 billion draft feeding a 120 billion target model. Typically it’s a percent or less. It’s a much smaller model. For example, in the Kimi K3 work that we’re doing, we would use a 4 billion drafter for a 3 trillion model. That’s a very small draft compared to the target model.
And the draft would be able to get many tokens right, and it’s always exactly the same as what the GPU would output because it’s always verified by the GPU. If at all the drafter makes a mistake, the GPU tells it what the mistake is and then the drafter corrects and continues from there. So there’s no difference in the output tokens that the GPU produces compared to suppose it was not running any speculative decoding. Using these rejection sampling approaches, we make sure it’s exactly the same accuracy. We don’t lose any accuracy.
And what is the bandwidth benefit you get by using this drafter-verifier approach rather than just running decode on a fast GPU?
Sudeep: The fast GPU would be generating one token at a time. And if you were to get let’s say five tokens correct on the average—let’s say the drafter produces eight tokens or 16 tokens and feeds that in a chunk to the GPU and on the average, let’s say we get five out of eight correct—you would get much better speed-ups. So we were able to demonstrate three and a half to four X speed-ups on the GPT-J model.
There are two ways you could use it. One is you could get to regimes you could not get with a GPU. In the work we did with Gimlet, for example, we could get to 1600 tokens per second per user. Something that the GPUs could not get to. The GPUs could only get to say 400 tokens per second per user. So we get this 4X higher interactivity. Another use case is for the same interactivity, you can get much higher throughput. So both modes are possible. And it’s up to the user how they want to trade off throughput versus interactivity. We essentially shift the whole Pareto curve to the right.
You did mention two other forms of disaggregation, which is like attention feed-forward disaggregation and then just disaggregating the prefill and decode. The broad idea is that even the process of inference has many ways to break up the problem and you could use SRAM-based inferencing for parts of them where it really benefits the most. All of those things, just like the speculative decoding example you gave, should speed up the inference and move the Pareto frontier of the interactivity curve in a good way. So these are all ways you can use the SRAM approach. This is a very good setting the stage for what comes next. But before we do that, we have to address what the LPDDR on the card is meant to do.
Sudeep: In the performance versus TCO curves, you can have super-fast these SRAM chips like d-Matrix or Cerebras or Groq. They’re super fast, but you sometimes require a very large footprint. That’s at the top of the tier. Then there’s the mid-tier, which is essentially all of the GPUs, and that’s where the bulk of the market is. And then there is another tier where you can store large models very efficiently on LPDDR. But you may not get the interactivity advantages of the GPU or of course the SRAM. But you could use that to reduce cost. You could use that to shrink the number of cards because you can pack a lot of memory on one card.
For example, the card we’re showing, you can put 250 gigabytes and this is commodity LPDDR. The same LPDDR that is found in cell phones and it doesn’t require three times as many wafers to produce the same bits as HBM. So it’s much cheaper because of that. So we cover both one and three. Obviously we don’t use any HBM on Corsair, we don’t touch two, but we cover the other two.
Raptor and 3D Stacked DRAM
So you have one approach to go really fast and another approach to go really slow. The really slow approach gives you more capacity, but not as much speed and that’s useful for some workloads. Maybe like offline batching, you could process all your day’s work at night and then next morning you can come back to inference results. You can run those cheaply on a widely available DRAM. Although I don’t even know what that means anymore. Is DRAM even widely available? There’s no such thing called cheap DRAM or widely available DRAM anymore. But anyway, it’s DRAM. It’s not HBM, which is even harder to get a hold of possibly. So it serves two different areas of high speed or high capacity. Because you don’t have HBM, you don’t have the medium-speed, medium-capacity tier, but you have the other two extremes. This is good. I think the next step we should go into is actually the meat and potatoes of all of the d-Matrix stuff that’s been going on at Hot Chips, which brought all the excitement and was all over Twitter after your talk, which is basically 3D DRAM. And I think it solves this high-speed tier somehow by increasing capacity as well. But before all that, we should explain what 3D DRAM means in this context because there’s a lot of confusion as to whether this is actually 3D DRAM from like the transistor level. So we should set the stage of what actually 3D means in this situation.
Sudeep: Yes. This is our Pavehawk proof-of-concept silicon of 3D stacked DRAM. What we mean by 3D DRAM is an inference accelerator where one layer is logic, which is where the compute happens, and another layer is the memory, DRAM, and those are vertically stacked. So unlike HBM which sits 2D, where you have DRAM that sits next to a GPU, here we vertically stack compute and memory.
This is helpful because now we can discuss basically why we had to stack DRAM on top of logic. And I have maybe several questions around this. The first one is, we will directly be able to increase capacity because DRAM is about an order of magnitude better bit density than SRAM is. So immediately you will go from the hundreds of megabytes per chip. In your case, you have multiple chips that give you 2GB of SRAM in the Corsair, right? But now, what kind of capacity do you get by just putting one chip of DRAM on top of your logic chip?
Sudeep: We get 32GB of DRAM. In the specific construction, we have DRAM at the bottom and logic at the top. And this is what we mean by 3D DRAM. You can see the cross-section on the right where they’re connected using micro-bumps. And the power to the logic die is delivered through the TSVs. The logic die itself in this case is a TSMC N4 logic die. And the bottom die is a custom-designed DRAM. And we’re using 36-micron face-to-face stacking. So going back to your question, Vic, we can pack 32GB of capacity at 100 terabytes per second of memory bandwidth.
Essentially, we got 16 times the capacity of SRAM, while we retained SRAM bandwidth. And in this particular demonstration, we measured 0.37 picojoule per bit because the bits only go down and traverse very short distances as opposed to HBM where it has to go down the entire stack, and then it has to go sideways, and then it has to traverse tens of millimeters on the logic die. All of that power adds up and you will end up burning about two to three picojoules, 2.5 picojoules to be exact on the HBM itself, HBM4 itself. But another two and a half to three picojoules on the GPU. So that five picojoules became 0.37 picojoules. And that’s the big advantage of this technology. For the same power, you can deliver a lot more bandwidth—in this case, 6X bandwidth—by using this technology.
So to put that in perspective, if you took your previous Corsair card, which had 2GB of SRAM, you need 16 PCIe cards, which you probably have to put in across two different servers. You can put only eight PCIe cards in a single server of Corsair. And then you’re going to have two full servers to get 32GB of SRAM. Now, Corsair required all that, while Raptor, just by putting logic on top of DRAM, you ended up getting basically two servers’ worth of capacity while retaining the bandwidth that you had in SRAM. So you had free lunch and you ate it too. Yes. This is very rare in engineering. That’s great. And the energy efficiency too. The distance that the electrical signals have to cross between the compute chip and the DRAM chip are so close by that you don’t spend as much energy shuttling bits between memory and logic as you would in HBM. Now, you have a micro-bump that is like, what was the size of that thing? 36 micron pitch. It’s a pitch, but the height of the bump itself is like tens of microns, right? So instead of going millimeters, you’re going tens of microns, so the energy required to shuttle bits is now 1/10th of what you would have to do with HBM. Or 1/20th even based on your five picojoules per bit number.
Sudeep: Exactly right. Because vertically, of course, you have to make sure that you’re not going to move a lot of data on the logic die. So we custom-designed and pitch-matched the compute engines directly above the DRAM banks, so we call it micro-banks, and we design a channel such that three micro-banks fit underneath the compute engine, and they’re pitch-matched. You have to co-design the compute and the DRAM.
Stacking Order and Future Bonding
Is there a specific reason you chose to put compute above the memory? I think there are two ways to do this. There is two options. One, you could have the compute chip on the bottom and put the DRAM chip on top, which seems to be kind of the approach that Cerebras wants to do at wafer scale or Qualcomm wants to do with their high-bandwidth compute where you put DRAM on top of logic. The other approach, which is what d-Matrix has, at least on the Raptor version of this, is logic that is sitting on top of DRAM. So, there are definitely some trade-offs here that we need to talk about. And why you chose this approach.
Sudeep: Yes. You can pick your poison. On the left picture here, you see the logic on the bottom, the DRAM on the top. And of course, since the liquid cooling cold plates are on the top, the heat has to conduct through the DRAMs. And the logic is the hottest die here and it has to traverse the DRAM. So the DRAM temperature will rise and thermally this is not as good a solution as one on the right. On the one on the right, the heat sink is right on top of the logic die because the logic is on the top. It doesn’t have to conduct through the DRAM. So the DRAMs don’t see as much of a temperature gradient as on the left. So from a thermal standpoint, the solution to the right is much better than the one on the left. But from a power delivery standpoint, the solution on the left is better than on the one on the right. So those are the fundamental trade-offs.
Yeah, so putting logic on top of DRAM means that you can just put the cold plate right on top of the logic chip and extract heat before it upsets the DRAM die too much. DRAM really doesn’t like heat, does it? And the problem with that approach, the poison you have chosen here is you have to deliver power to the logic chip through the DRAM die, which is a power delivery challenge because you want to make sure that there is no power drop across these vias that are going through the DRAM die to the logic die. So, clearly you decided that thermal was a bigger problem to fix early on.
Sudeep: That’s correct. For a one-high stack, we found that we could in all our modeling, we could reliably cool the DRAM to be less than 95°C for power densities of the order of 0.5 to 0.7 watts per millimeter square. And that’s the power density of our logic die. So we could solve both the power delivery problem and the thermal problem using this approach.
In the future, I know right now you have micro-bumps connecting these things together, but in the future, do you think that going to maybe hybrid bonding will help in the bandwidth aspect because the pitch can get closer than 36 microns? But do you think that helps the thermal aspect if you go to hybrid bonding in the future?
Sudeep: Yes, it does help the thermal aspects, especially if you do multi-high stacking. But the fundamental problem is still reducing the energy consumption on the DRAM. Because you still have some routing on the DRAM and it still takes like in our case, 0.37 picojoule per bit. So you can pack with wafer-on-wafer, you can get a lot more I/O than say 100 terabytes per second, right? It could be 400 terabytes per second. But if you don’t reduce that I/O power from say 0.37 picojoule to 0.1, the total power that you’ll dissipate just in the interconnect is so high that that is no longer the limiting factor.
Yeah, that’s a very good point and to put some more light on that, the way you can easily calculate how much power this kind of a chip at a given memory bandwidth will consume is if you take the bandwidth, let us say 100 terabytes per second, and you multiply it by the number of picojoules per bit, it automatically spits out the power consumption in watts. So if you go four times higher in the memory bandwidth by doing some techniques, you want to be sure that you reduce the energy per bit four times to keep the same level of power dissipation at the chip level. Otherwise, what ends up happening is if you boost the bandwidth, but you’re still spending energy shuttling bits, you’re just shuttling more bits and that’s going to cost you in terms of power.
Sudeep: Yes, so 100 terabytes, so you convert it to terabits. So 100 * 8 * 0.37, that I think is about 280 watts. Just in the memory subsystem to move those bits into the compute unit. And now if you start putting stacking more, maybe that number will go up. If the number, let’s say, even remains the same, but if you want four times the bandwidth, then that’ll end up burning upwards of 1,000 watts. So we have to first figure out from an accelerator standpoint, what’s the bandwidth we desire, how much power we want to spend on the memory subsystem. Because there’s still compute you have to spend power on, interconnect you have to spend power on. Typical rule of thumb we use is 60% on compute, 30% on memory, 10% on interconnect. And then figure out what technologies can help you get there.
Building Racks with Nvidia
That’s fascinating. What comes next after this? The really exciting part of the news that recently came out from d-Matrix is the use of the Raptor chips in the Nvidia NVLink Fusion ecosystem. What’s the idea and what are you thinking going into putting Raptor chips into the ecosystem? What is the benefit of that? And really how easy is it to integrate Raptor chips into the Nvidia ecosystem?
Sudeep: The rack is our product. At the end of the day, like you’d pointed out, models are very large and let’s take an example like Kimi K3. Kimi K3 is the largest open-source model out there, 2.8 tera. It’s a huge model. And it’s got 1 million context. Obviously, it won’t fit in one chip, GPU or otherwise. So the whole idea is to put many of these chips, in our case, 144 of these R4 packages. The Raptor package looks exactly like this. There are four chiplets in a package. And there are 144 of those packages inside a single rack. And for that, we chose the Vera Rubin rack. And we use NV switches. So we have 18 compute trays and nine switch trays. And each compute tray has eight of these packages. So that’s how we get 18 * 8, 144. So we have 144 Raptors inside a VR rack, Vera Rubin rack.
What we do is we keep the Vera CPU the same, all of the networking, the CX and the Bluefield networking the same, and replace the Rubin GPU with Raptors. And instead of four Rubins, we have eight Raptor packages. So a sum total of 144 chips that’s all-to-all connected. And this rack then can talk to a Rubin rack because they use the same networking technology.
So you just have to pull out all the Rubin GPUs, there’ll be like four of them, put in like eight Raptor chips. You keep the Vera CPU and you build a rack with 18 computer trays. Each one has eight Raptor chips, that’s 144 chips. Each one, what is the total capacity of the 3D DRAM in a single rack?
Sudeep: 2.3 terabytes of capacity.
So that’s pretty big to hold an entirely large model all by itself actually, if you wanted to do that.
Sudeep: Correct. We were showing at Hot Chips that we could hold the Kimi K3 model at 4-bit precision and have sufficient space for many users’ KB cache space and we could hit 1,000 tokens per second per user. It’s a performance metric that you can’t achieve with traditional memories. The 3D DRAM, because it’s such high bandwidth, you can get these levels of interactivity at high batch all in one rack. In a single rack, we can get to a 3T model at 1 million context at 1,000 tokens per second per user on Kimi K3. And on GLM 5.2, we could get to 3K. So these are really fast decode speeds.
And because in the Nvidia racks, you can do multiple racks, there is an NVL 576 option where you have four racks and the four racks are connected using optics, optical scale-up. And using that now, you have not just 2.3 terabytes of capacity, but 2.3 terabytes times four. So 9.2 terabytes of capacity, which is sufficient to, at FP4 precision, serve even a 10T model. So we can serve out the whole frontier at unprecedented speeds because of our rack-level, rack-scale solution as well as the fundamental technology that delivers much higher bandwidth.
So you’re talking about basically putting four racks of Raptor chips, each with 144. So that would make it 576. Hook them all with optics or any form of very high-speed interconnect, and then now you have a large scale-up domain with basically nearly 10 terabytes of SRAM-like bandwidth, but with 3D DRAM capacity, which can serve any large frontier model of the future at ridiculous speeds.
Sudeep: Exactly.
Software and The Future
Seems like a winner. How does the whole software stack work though? So if you put in a rack of Raptor chips next to an Nvidia rack running GPUs, all of that stuff runs on CUDA. How does the software work? How do you do the disaggregation?
Sudeep: With Dynamo and frameworks like that would handle the disaggregation. sglang or vLLM, any of these frameworks would partition the work even today between say a pre-fill cluster and a decode cluster. Same thing happens here. But underneath that, when it comes to generating code for our chips, we have our own software stack. We provide something called a Model Builder SDK. And the Model Builder SDK essentially we expose a set of primitive kernels—MatMul, SIMD, various shapes and so on—and a way to stitch together models using those primitive kernels. And a programmer can choose to write new kernels if they want to, because we expose the ISA of the chip, or just to take these building blocks and compose models. And we provide a lot of example models. And the AI kernel generation tools have gotten so good these days that using these examples and using these agentic harnesses, you can quickly come up with a new model and perform a new model in a couple of days.
So that’s the lowering stack. And then we have our own execution stack where the device runtime, there’s a runtime that runs on our chip. And that runtime runs on the embedded RISC-V processors that are part of each of those chiplets. It’s the same on Corsair, it’s the same on Raptor. And everything essentially is targeted on that RISC-V.
I see. It’s fascinating. What comes next after this? Clearly if you can have one DRAM stack, what are you going to try next? Put two, three, four? What’s the roadmap on that?
Sudeep: We are trying to build up to a four-high stack. On Lightning, which is our next product, we have our logic on the bottom and then four DRAM on the top, and we’re trying to build up to that. Now, we have obviously lots of challenges in terms of thermals, in terms of the bandwidth we can deliver, in terms of the availability of hybrid bonding. As you know, hybrid bonding of DRAM and logic hasn’t yet been successful, mainly because of contamination or warpage. These are fab-grade processes that you cannot take a wafer into a clean room and contaminate, for example, a logic fab. So all of these are challenges that need to get worked through.
But more importantly, from an architectural standpoint, if you cannot test any of these things and you’re just going to hook this up, how are we going to solve this problem? How are we solving the yield problem? How are we solving redundancy problems? If you look at even Raptor, we do refreshes that are eight times faster, 4-millisecond refreshes compared to 32-millisecond refreshes, precisely because of these thermal concerns. And now, what is the solution for a multi-high stack? So stay tuned. We have solutions to many of these problems, but it’s definitely a work in progress.
I think to me that the inference world right now is kind of the wild west. Like making training chips seems to be like there’s one or two methods to do it. Most of them are GPU-based. But inference is all over the place. You’ve got wafer-scale engines, you’ve got SRAM-only accelerators, you have SRAM plus LPDDR, and then now you have 3D DRAM, and then you’ve got other companies who use all three tiers. Everybody has their own approach. So in the future, say a year from now, do you see any kind of a convergence into what is the right architecture we should be doing? What is your take on that?
Sudeep: It is what we’ve been discussing. To us, it’s fairly obvious that the densest memory today is DRAM. And because it’s 1T1C, you can get 40, 50 megabytes per millimeter square, versus four or five on SRAM. So you get much denser storage and you can get the bandwidth by stacking. So with this heterogeneous integration approach where you stack, the benefits are so obvious and specifically the metrics we called out, like on bandwidth and power, is so compelling that it’s only a question of when, not if. So we feel like in five years, this is the wave of the future. Everyone will be doing 3D stacked DRAM. Now, there are engineering challenges to solve along the way and we’ll all figure out. There’s a lot of problems to solve and we’ll figure out how to solve them, but we believe this is the end state of inference.
That’s amazing. I’m only going to leave you with one contest that I think somebody in the future is going to come over and say, hey, why don’t we stack high-bandwidth flash on top of logic? That’ll give us even more capacity, right? And then you open up a whole new world of problems and then we can all work on it for another five years. This is great.
Sudeep: Yes, exactly. Talking about wild west, it’s a really exciting space. There are so many new ideas, there are so many avenues to explore. I’m sure creative people will find new ways to solve problems. It’s just a really, really exciting time to be in this space.
Yes, absolutely. Thank you so much for doing this podcast with us. As always, whether it’s your talks or meeting you in person or on the podcast, I always learn something new. So I hope our listeners learned a whole lot of stuff about d-Matrix and how everything has evolved from your initial analog in-memory compute days, far before even GPT was a thing. You’ve started building something for inference, which is clearly well over half a decade out, and you have working silicon that is now going into racks, and it’s linking up to Nvidia’s ecosystem and in the future, you’re going to have even more memory per chip and so you can build even more models and bigger things and smarter things. So that’s all very exciting. So thank you very much.
Sudeep: Thank you, Vic. I’ve always been a fan. I enjoy your writings and I enjoy your podcasts. It’s great to be here and talk about what we love to do. So, thank you for having us.
All right, that’s it for this episode of the Semi Doped podcast. If you enjoyed it, leave us a five-star review on Apple Podcasts. It’s very important. Rankings are always important. We always want to rank higher than somebody else. So, get that and you can always watch this on YouTube or any podcast platform of your choice. And if you enjoy it, share it with a friend. And we also have a daily newsletter on daily.semidoped.com where we keep track of all of the daily news. So, check that out as well. Catch you on the next one.
