Vik Sekar and Val Bercovici dive into the rapidly evolving landscape of AI memory and storage. They explore how Weka’s Augmented Memory Grid (AMG) leverages high-speed networking to surpass traditional DRAM performance. The discussion covers the critical role of KV cache optimization, the impact of DeepSeek V4’s pricing, and the future of NAND flash tiers.
Things we cover:
* Weka’s Augmented Memory Grid (AMG)
* NVLink vs. PCIe for memory bandwidth
* KV cache optimization and DeepSeek V4
* NAND flash tiers (SLC, TLC, QLC)
* The future of CXL and High Bandwidth Flash
* AI tokenomics and the software industry
This podcast is lightly edited for clarity.
Welcome: The Ever-Changing Memory Landscape
Vik: Welcome to another Semi Doped podcast. I’m Vik from Vik’s Newsletter and with me is Val Bercovici, Chief AI Officer from Weka. Weka is an AI data and memory infrastructure company. Val has been on Semi Doped before and a few months ago we spoke about context memory storage right after Nvidia announced their platform.
If you want to check out that whole conversation, it’s right in the back catalog and we’ll try to link it up here as well. But today’s conversation is more about more recent developments in the world of memory and storage and where things are going in AI inference. Val, thanks for being on the podcast.
Val: So much fun to be back, Vik, and as we predicted, the only constant is change. So, lots of updates to talk about since I was last on.
Vik: Awesome. Isn’t that so exciting? We have things that are constantly changing and since we spoke, what was it, a few months ago? Seems like everything has changed and you were mentioning before the show, seems like 10 years have gone by in AI land.
Val: Exactly. In fact, I don’t know if I made this prediction last time. I expected a mythos fable moment at the end of the year and it happened in like what, May or April, I forget exactly when. So, things are definitely happening faster than we thought.
Vik: That’s amazing. Yeah, it’s really picking up now. And memory has reached an all-time high. It’s become extremely expensive. I think a lot of people who are deploying AI hardware are finding ways now to make the best of what they’ve got. Increase utilization rates, pool stuff, offload to different tiers, whether it’s other kinds of RAM, not just HBM, or go to NAND flash. So there are so many techniques that have sprouted up in a matter of mere months because we have to do something about the memory situation.
Val: Exactly. I would say even one of the, I wouldn’t say it’s the simplest, but one of the most obvious and coarse-grained techniques in my mind has been model routing. If you’re not always using the most memory hungry model for every prompt, especially in an agent swarm with hundreds or thousands of turns, and you’re intelligently routing requests to medium and small models, implicitly there, you’ve got lower memory requirements. So that’s kind of an easy button, if you will, for reducing memory in aggregate. But then we get obviously much more specific here with regards to each model’s memory requirements, working memory, KV cache, and so forth.
Vik: Yeah, exactly. So that’s what we were talking about in some sense last time because the KV cache started getting so big when the agents arrived on the scene that you couldn’t keep it on expensive HBM because it’s a very limited resource and it has to be preserved and used judiciously. So there was a lot of talk about offloading to DRAM. And then as agents started working more and more and having greater context length, they started talking about offloading to context memory storage, which is what was this whole rack, which Nvidia calls what was that? STX, right?
Val: Yeah, the new storage platform is STX. The biggest use case of STX is going to be context memory storage, CMX, context memory extensions in Nvidia terminology. And that is, within the STX hardware framework, hardware and software framework.
Weka’s AMG: Faster Than DRAM
Vik: Awesome. So yeah, I know Weka has hardware. We spoke about this last time too, where you’ve got Weka’s AMG, which provides a lot of storage directly to the GPU via a very fast network connection. And the speed that you get out of it is really good, comparable to something faster than NAND but slower than DRAM, right? So that’s where we were.
Val: Let’s double click on that because it’s actually different than that even. So the standard positioning is you can be faster than NAND with higher bandwidth connections, faster than local storage effectively. But in a standard hierarchy, you could slot in a slower than DRAM. However, depending on your architecture and whether you can take advantage of the full line rate network performance or something like NVLink, the way Weka does, you can be faster than DRAM.
So if you actually take a look at the number of PCI lanes that we have in these wonderful scale-up domains with NVLink on Nvidia servers in particular, there are far more PCI lanes, like 128, I think, in NVLink than there are PCI lanes from the CPU to DRAM, which is only 32 on the motherboard. So there’s more, this is one of the cool architectural leverage points of Weka is we recognized at company founding inception about 13, 14 years ago, for high performance computing, which is what AI factories are, there is more bandwidth on the network than on the motherboard. And PCI is a bottleneck, it’s not an accelerant.
And this is still something that I find a lot of systems architects and developers struggle with. They assume nothing is faster than what I can access on the motherboard. And they design entire database hierarchies, entire application latency profiles based on that invalid assumption today, because the network is faster than the motherboard in a GPU network, in a high performance network. And so yes, I think the memory tiering can have whatever you connect on the other side of that high bandwidth network, an NVLink style network, east-west, compute memory network, NIC network, it goes by many names. Whatever you connect on the other side of that can be faster than DRAM. And the way Weka connects, optionally, we don’t require that network, but we certainly leverage it maximally. But the way Weka connects, we’re faster than DRAM on a raw bandwidth basis.
Another confusing part is the way CPU and GPU memory transfers, KV transfers, for example, and others, operate on most of these servers, GPU servers. Depending on the generation, Hopper has something called a bounce buffer between the GPU and the CPU. Blackwell introduced, especially with Grace Blackwell, more sophisticated chip-to-chip interconnects that are faster between GPU and CPU transfers. Vera Rubin yet again continues to advance that with higher bandwidth, far more scalable memory transfers between GPUs and CPUs, but you can still be faster than DRAM on these servers because of NVLink. There’s a ratio in terms of NVLink bandwidth relative to memory bandwidth relative to front-end network bandwidth. And as long as you can be line rate on NVLink, you can be faster than DRAM, which is counterintuitive, but the math proves out.
Vik: That’s awesome. Okay, that’s great because just to recap what we are doing here is that the networking on modern hardware is so fast that when you hook up even storage using that, you can get bandwidths that are faster than DRAM because motherboards aren’t the fastest. And that is an assumption that a lot of people are making right now. But instead, if you view the NVLink as a very high bandwidth interconnect, you can actually bring storage to operate with much higher capacity, but at speeds that are better than DRAM speeds.
Val: Yeah. And these networks are really critical right now. We’ve seen a rise in interest in networks, not just with co-packaged optics, CPO, but what Google does with their Taurus network. AMD obviously with Pensando and that line of networking that they’re going to continue to improve with Helios. But just what Mellanox has done with NVLink has been wonderful. And it’s not just NVLink, right? Mellanox offers CX7, CX8, CX9 network adapters, Bluefield 3s and 4s. There’s multiple varieties of Bluefield 4s. And so we’re not picky at least at Weka with regards to where the bandwidth comes from. As long as systems architects configure the bandwidth they want for the memory tiering they want, Weka can support it and continue to deliver these radical economic benefits in terms of more tokens per second, more concurrent users per second, which is really hitting the market’s sweet spot right now of agent swarms.
Inference vs. Training & KV Cache Dynamics
Vik: Awesome. So since we last spoke, Weka right now shouldn’t be viewed so much as a storage company anymore. Because in the AI era, because of what solutions Weka provides, I think it’s far more than that. We were speaking a little bit about it before the show. Do you want to double click a little bit more into why Weka should be seen more as a memory company and a company that does AI data infrastructure instead?
Val: Yeah, very much. Two of our industry partners, if you will, one is Groq with a Q that Nvidia acquired last December for $20 billion and a non-exclusive license to their technology and most of their founders and engineers. And then of course, a very popular and successful Cerebras IPO, were clear indicators that the inference market, the inference infrastructure market is very distinct, very different than the training market that Nvidia dominates with GPUs.
And even Google themselves, for me, for some reason, this is like the clearest example. Google has had seven generations of TPU years that were applied both to training and inference, right? And radically different markets back then when TPU 1, 2, 3, etc, through 7 was released. With TPU 8, which they announced just a few months ago, they were very explicit as saying, there is now a TPU 8 for training, but there’s a completely different TPU 8i for inference. So that’s just another example that inference is different, inference infrastructure is different. It’s great when you can leverage the same infrastructure for training and inference. There’s nothing wrong with that, but understand that, whether it’s really cool companies coming back out of stealth like Etched, or Cerebras or all sorts of other LPU-based, ASIC-based, SRAM-based companies, inference will continue to diversify in terms of infrastructure and diversify away from training.
So Weka is very much following in that trend. Weka has a great legacy of high-performance computing HPC storage, but the new tagline for the company, data and memory infrastructure, reflects the fact that at its first principles raw basis, inference is not storage-centric. Inference is a little bit compute-centric as we know, for pre-fill and extremely memory-bound for decode and feed forward and so forth. And that’s really where the Weka Augmented Memory Grid product line fits in, is it’s very much you can use it as an inference-only product without any Weka storage whatsoever. It just happens to be backed by NAND at the bandwidth of memory, and as we just discussed, when that particular context memory network happens to be large, then it’s actually more bandwidth than DRAM.
Vik: Yeah, that’s awesome. So, depending on the kind of hardware that is running now, you mentioned that Cerebras is one, and the whole inference landscape really seems like there’s no one right way to do inference. You could run it all on SRAM, you could run it with LPUs, which is also SRAM. Then you could run it with a combination of HBM and SRAM. Or if you look at SambaNova, they use all three. So, it’s really breaking up all over the place and so, how do all these compare in terms of performance or where does storage play a role into this whole thing?
Val: So, there’s a memory hierarchy, and it’s interesting now because when we last talked about this memory hierarchy, it’s funny, we talk about it as if it’s been around forever, but it’s really only about a year old. But a fairly well-established four-tier hierarchy that’s going to have to change now. And Nvidia’s Dynamo team has done a really good job of documenting this. They even label it G1, the top tier in the memory hierarchy is for high-bandwidth memory. G2 is for CPU DRAM, but it also goes by LPDDR and SoCAM, but it’s, for the uninitiated, it’s all the same thing. And then so that’s G2. G3 is what’s called local storage and sometimes it’s called the rack SSD that’s in the servers. And then the G4 tier is remote storage. Often it’s NFS or S3 type storage layers.
And the one thing to really not maybe confuse about this memory hierarchy is there’s no smooth graduation from HBM to DRAM to storage, local or remote storage. These are very, very rough, sort of jagged transitions, almost a Grand Canyon sometimes between them, because we’re talking about orders of magnitude higher latency between those memory tiers, orders of magnitude different bandwidth between those memory tiers. And you very quickly reach these cliffs where, if you’re okay providing one token or 10 tokens per second to a user, and you’re okay with latencies, time to first token of like 10 seconds and end-to-end latency of hours where they should be minutes, then you don’t have a problem. But in the real world, no one tolerates five or 10 tokens per second output. The human eye demands, human attention span demands about 35 to 50 tokens per second of output minimum. And agents, of course, at machine speed will take thousands of tokens per second of output in a multi-turn agent swarm.
So, SLOs, service level objectives matter. And that’s where we quickly find right now that even though there’s a lot of vendors talking about memory tiering, including NAND and storage, in a KV cache hierarchy, it’s kind of irrelevant. Most of the benchmarks we see out there today that include storage in a KV cache tiering hierarchy are workloads that you could just run on a DGX Spark or a Mac Studio today. You don’t even need a big server to run some of these smaller models with small context windows and single-turn chat sessions or two to three turn agent sessions that are not representative of reality.
When you’re running real-world workloads today, and SemiAnalysis is actually working on updates to Inference X that they’ve nicknamed Agent X after my suggestion actually. So Agent X when it comes out soon, I’m not going to steal their thunder, but when that comes out soon, that’s going to reflect something called an input sequence length, an ISL, which is the context window, of much larger than 8K, which has been the limit in the past. Hundreds of K, they themselves have published analysis of the fact that I think the median traces for coding agents now are 2, 300K tokens, because of course, we’re in the era since we last spoke of million token context windows instead of 100K or 200K token context window limits.
So, when you reflect the reality of large models, trillion parameter models or models in that class, MiniMax is in that class even though it’s not quite a trillion parameters, but Kimi is and GLM 5.12, the new hotness is and so forth. When you reflect large models with large multi-turn sessions with high concurrency, the way Claude Code, Open Code, Codex, actual Pi, Hermes, Open Claude, actual agents work, then you end up with a radically different workload profile. And you on the one hand, the models have gotten so much better in terms of KV cache consumption individually, right? I use the number I think I remember of for every 100K tokens without optimized compression like TurboQuant, without optimized compression like sliding window attention and others from DeepSeek V4, you were up to 50 gigabytes of KV cache usage for that 100K tokens. That’s been reduced on a unit basis right now with TurboQuant, with optimizations from DeepSeek and others, to about 5 gig of instead of 50 gigs. So, that 90% reduction is great. But what’s happened on the other side? Context windows 10X. And then agent swarms 10X, 100X the number of concurrent sessions. And on and on and on, we’ve gone multimodal. And so the net effect we always seem to see is that for pretty much every 100X reduction in unit KV cache size, there’s a 10,000X increase in consumption of KV cache. So, we’re always looking about that 100X more token volume, and it’s reflected in the pricing pushback we’re seeing by enterprises today. Maybe they’re not paying 100X times more than they expected, but they’re paying a lot more than they expected because of that token volume.
Vik: Yeah, yeah. So this is great, right? Because clearly KV cache optimizations are on the way. TurboQuant was one which scared the daylights out of the market. Everybody thought memory is dead, HBM is dead, everything is dead. But it turns out that the quantization is a good thing because we only need like you said 1/10th or 90% lower KV cache storage for 100K tokens. So, 100K tokens was taking 50 gigabytes, now it takes 5 gigabytes. But this is great because which means with our existing models, we can run far more agents and do a lot more work. And so, the demand really never went away. In fact, it got more, right? It increased.
Val: Yeah, yeah. In fact, you could almost make a meme out of this. That 50 gig that went down to 5 gig went right back up to 50 gig because of context window size went from 100K to a million. So, there’s a give and take here and the net result is just always more memory demand.
Vik: So, just because KV cache is being compressed like this, do you see that it’s going to continue to be so? Do you think that we’ll go to maybe one gig per 100K tokens in the future? And what does that mean for KV cache? Because if you say like it dropped the usage by quantization techniques, and then we brought it back up because of agents. I guess we don’t have a net increase. Is that something that the memory and the storage industry should worry about? Because I don’t know if the increase in the usage is exceeding the benefit coming in from the quantization.
Val: That cliché, “this time it’s different,” applies here too, right? Every time there’s this massive new innovation in KV cache compression, it’s not different. Jevons paradox keeps kicking in over and over again. And we’re seeing that, that higher increase in demand every time there’s a reduction in the unit consumption. And so, for me, let’s look at context windows. We’re not stopping at 1 million. It won’t be long before we’re two and five and 10 million maybe by the end of this year. Everybody wants more context. And we’re going again, longer horizon in our agents. We’re giving agents far more ambitious goals, being encouraged to by the agent and model providers. The goals now span hours and days. Soon, I think some agents will run weeks quite regularly. In fact, for cyber security, which we can get into, they run forever, right? You’ve got to run the security operation center persistent blue agent swarms forever now.
So, we’re seeing longer multi-turn horizons. Of course, we’re seeing more parallelism, more concurrent subtasks. Claude workflows was one sort of feature introduction that became very popular to just solve a problem in parallel 10 times faster. And then again, we’re going multimodal. So it’s not just language, but we’re inserting more video and audio frames and image frames into these agents and it all just results in a continuous explosion of overall token demand and overall KV cache consumption.
DeepSeek V4 & The Cache Hit Revolution
Vik: Yeah, that’s awesome. Okay, so now that we’ve established that this is only going to get more from here. So, NAND and DRAM or whatever form you would want to look at it as, pooled, not pooled, we’re going to need more of this because we’re only going to run more of these things, so we’re going to need more. So, that doesn’t spell any decline or leveling off in the near term because context windows are going to get longer, agents are going to run longer, and more agents are going to run per task because you can have parallel ones run.
But I think one interesting thing that you mentioned earlier was DeepSeek’s optimizations because when DeepSeek V4 came out, there was a lot of emphasis on an SSD first approach to inferencing, which I think really helped in terms of their token pricing, especially when you end up hitting cache. And they showed that for some workloads, you could hit like 95% cache hits. And what happens then is that as long as you keep hitting KV cache, you already have the tokens stored in a high bandwidth network connection to an SSD, which means now you can basically get free. It’s free now because their token pricing is so low for cache hits. It’s insane. How is this playing out into other models or are other models also hitting cache hits at this level or what do you think of that?
Val: Yeah, I’m so on the one hand, I’m glad you noticed that because we really want to emphasize this key point now. The majority of token volume is agent swarms by far, 80, 90%. It’s no longer chat sessions. So chat sessions are almost irrelevant to the conversation from an infrastructure perspective. And agents, it’s interesting, even the Claude 3.5 Sonnet announcement just the other day, right? That everyone looked, even Anthropic, interestingly enough said, here’s the new pricing for input tokens and here’s the new pricing for output tokens and it’s a promotional price. And the reason why I don’t even quote the numbers, it’s really irrelevant because of all the tokens used by agents, most of them are cache read tokens. And some of them with Anthropic’s particular, unique pricing models, you have to pay for cache writes as well to benefit from cache reads. But we should only be talking about cache read pricing. The other pricing is a rounding error.
And DeepSeek’s pricing reflects that, right? They’re dramatically new, inserting two zeros, basically after the decimal point for price per million cache read tokens, really is a shock to the industry. Not so much the memory industry, it’s a shock to the other models and the other inference providers. But the very important asterisk there is that pricing is only available out of China, right? You have to use DeepSeek hosted, the rumor is I think a Mongolian data center with very, very low energy costs, but also some secret sauce that DeepSeek has with regards to how they’ve integrated version 4 Flash and Pro with the fat high flyer file system, the 3FS file system, parts of which is open source. They famously open sourced that last year and wrote some wonderful papers and blogs about it. Curiously, unless I’ve missed something, they haven’t updated those blogs with what definitely are some new innovations and improvements that factor into that dramatically low pricing.
The net of it though, I think VentureBeat did some math which I was quoted on earlier on is it’s about 87 times lower cache read pricing from China than the same model, V4 Pro or V4 Flash hosted in Singapore or hosted in the US or Europe. And so the real conversation is yes, DeepSeek has completely set the bar for global pricing for people that are able to inference out of China, but for people that can’t inference out of China, it’s still by far the best pricing, but there’s a massive margin opportunity for open weights inference providers to differentiate on cache read pricing because you may not be able to match DeepSeek’s subsidized pricing, but you can still leverage their innovations, their sliding window attention, hybrid compress attention, HCA and so forth. You can still leverage those innovations and offer very important reductions in token cache read pricing and really drive effective, aggregate agent pricing down because that is the one pricing metric that dominates the cost of agents.
Vik: Yeah. So I have a couple of questions on this one. The first one is, why is it, it’s news to me that that pricing comes only out of China. So if I run a DeepSeek model out of, I don’t know, OpenRouter, I don’t have that pricing.
Val: OpenRouter is wonderful because it shows you the difference, right? In fact, for some reason, I don’t even know exactly what this signifies, but there’s a little slider button you have to slide to say “show ignored.” And I’m not exactly sure why they label the button that way, but when you click on that, it does expose the DeepSeek pricing out of China. And then you can compare, in fact, your agent, of course, your Hermes, your open client agent can compare side by side in real time the pricing for the exact same model, the exact same input pricing, the exact same output pricing, the exact same cache read pricing. And I should double check because I tend to stand that OpenRouter site quite a bit daily. I should check whether there’s more and more cache write pricing beginning to appear because I know for some models it has and it escapes me whether it’s appeared for DeepSeek providers as well. But you’re going to see some providers now differentiate not just on cache read pricing, which is a major point of differentiation, but also on cache write pricing.
Vik: Okay. In terms of cache read, in the real workloads that we’re running today, it could be agent workloads, right? Because like you were saying, any other kind of workloads like chatting and typing in, on the chat window is a negligible portion of the inference market today. So let’s just talk agentic workloads. How does somebody make sure that they have a cache hit rate of, I don’t know, 95% because that will drive the pricing down enormously and create meaningful differentiation like you were saying. And where is the industry right now in terms of cache hit rates?
Val: Yeah, you know what, we could spend a whole pod just on this question because it’s very opaque and a little bit confusing. So let me go through this because this comes up a lot in other pods as well. When you look at your agent dashboard, your Claude Code or just your Claude dashboard if you have both Code and Co-work and other Claude products or Open Claude, pick your dashboard, right? It’ll have a cache hit rate. That is a logical cache hit rate. That reflects the cache ability of the tokens in your agent swarms. And that is often very, very high. Like every agent pretty much has a cache ability of 95%, right? Because you are reusing a lot of context for agents, especially agent swarms and so forth.
However, the providers, the actual cache hit rate from the provider is not one to one the cache ability of your tokens, right? The effective cache hit rate is very much a function of the memory tiers you have. And everyone or by and large, most people in the first quarter of this year and some in the second quarter have had very finite fixed memory tiers. There’s only so much HBM that comes packaged on your GPUs. There’s only so much DRAM that’s on the GPU servers. And as I mentioned before, you can try and add storage tiers and KV offloading to storage, but they ruin your SLOs, so it’s really not that common in production for the popular models and popular token consumption.
And so the way to really understand where people actually have true effective cache hit rates that are as high, maybe not as high as the logical cache hit rates, but close, is in the pricing. So OpenRouter again is a great proxy for that because it’s the only way to introduce transparency to the real world cache hit rates. And even OpenRouter themselves, they publish actual by provider, by model, by provider, they publish some cache hit rates. And I like them because if you look every few minutes, they do change, right? Based on the actual token traffic of the moment. But even with some of the numbers I see there, I don’t think there’s a real effective infrastructure hardware memory cache rates, there’s some blend. But nevertheless, they’re much closer than what your own agent dashboard reflects. And yes, for me, I like to joke now, we’re seeing a lot of benchmaxing and a lot of the models right now, for example, like Swebench, everyone sort of trains for, so it’s no longer that relevant to benchmark, but DeepSeek is still a good benchmark. We see the same thing obviously when vendors sort of benchmark their KV offloading solutions. There’s not a lot of truth left in benchmarking. So my personal belief is that profit and loss, pricing, transparent pricing is the ultimate benchmark. And then you’re going to see Weka be more explicit in that space to do that, right? We want to continue to prove out our own advantages and we think that real world metrics reflected by pricing is probably the best way to actually benchmark a solution now as opposed to something in a lab.
Vik: You also mentioned that DeepSeek sliding window attention is something really to look into. What’s unique about that sliding window attention in DeepSeek?
Val: It’s actually, if you’ve ever paid attention to where compression of files in general has evolved from simple PK zip compression or simple block-based deduplication towards the similarity hashing algorithms and a combination of, sort of global similarity and global hashing and local hashing. The same thing is happening now with KV cache. The same concepts are being introduced at the token and the attention level. And sliding window is just that. It’s a way to take a look at just the more recent tokens and compress them as much as possible and just assume the tokens that your particular attention head hasn’t paid much attention to recently aren’t as relevant, aren’t as compressible. And it’s just cumulative, right? I think if you count it, there’s formally about five different attention mechanisms in DeepSeek V4 now, all simultaneously applied. And each one specializes in short-term compression, long-term compression, context relevant compression and so forth. And then that results is that impressive 90% real world reduction in KV cache usage per request. But again, the requests now are just coming faster and bigger. So it’s kind of necessary to support, in fact, I think the reason why they continue to innovate so aggressively on KV cache consumption is they want to support million token context windows and soon two and five and 10. And the only way to do that effectively is to keep being more and more efficient, intelligent, current, if you will, on how you consume KV cache.
Vik: Okay, I see. So, DeepSeek what it does, the sliding window attention idea is that you just keep the window of attention to what is most relevant right now and keep discarding what isn’t relevant anymore. And that saves you KV cache.
Val: Almost assume what’s been attended to before, that’s not being attended to now, has already been compressed to some extent. So now let’s really be aggressive on what we’re tending to right now and see where the opportunities to compress token memory, token attention and KV cache is.
NAND Flash Tiers: SLC, TLC, QLC
Vik: Okay. In terms of just NAND flash storage even, I wrote an article recently about what it really takes for an SSD to be AI ready. SSDs in the past were meant for a different use case really than what they’re being used for now. And one theory I have, which I want to run by you, is that NAND itself is now breaking up into several tiers. Like we always in the past wanted to get more capacity out of NAND. So the industry was trying to go from single level cells, which has lower capacity, but more endurance and faster write speeds. To going to QLC or, quad level cells on the other hand, where you can save, so basically store four bits or 16 different states in just one cell. So it really gives you a lot more capacity, but endurance is a concern and I’d say it’s harder to write to because you have to make sure you can differentiate between 16 states. How do you see these NAND tiers working out? Are single level cells making a comeback? How’s that working?
Val: Yeah, it is making a comeback. And so, just like we talked about earlier, training infrastructure used capacity NAND because you wrote a lot of training data infrequently and you read it very, very frequently. So that was ideal for QLC type approaches. Whereas inference is just fundamentally different, right? It’s really fundamentally memory-centric. So you’re really trying to make NAND appear more like memory and a lot less like storage. And memory doesn’t care whether you write or read at the same rate because the assumption is there’s no endurance issue, right? There’s a power issue, but there’s no endurance issue, a persistence issue, but no endurance issue with memory, with DRAM in particular or HBM. And that’s not true. So this is where NAND flash struggles to act more like true memory from a performance and a consumption perspective as opposed to just a capacity perspective. Again, this is something that Weka anticipated a long time ago.
And so you’re seeing that largely because of inference alone. Intel had this really interesting technology with Micron called Optane or 3D XPoint. And it’s a real shame. It’s a real shame that was a while. Oh my God, yeah. Today inference is the killer app for Optane. But it’s gone now, right? Based on different materials, phase change memory and so forth. And what has tried to replace it is SLC flash, right? If you have to, if you can’t optimize your writes, your KV cache writes, well enough, then you have to buffer that endurance issue, that drive wear issue with a more endurant form of NAND flash, which is a single layer cell SLC tier. And sometimes that can be complemented by QLC, so you buffer a lot of writes in SLC and then you de-stage them down to QLC later. Again, that adds complexity, that adds latency, there’s no free lunch there.
Another thing you can do, and this is something that Weka does, is kind of anticipate that you’re going to write to NAND flash a lot, you’re going to read to it a lot. Basically use NAND flash as opposed to just keep it for capacity. Yeah, right. And in that case, yes, what Weka has done is we’ve always amortized writes across a whole fabric of NVMe drives. My joke is there’s no S, there’s no storage in NVMe. It’s non-volatile memory express or memory extensions. And when you treat it like a true memory protocol and you amortize the writes across a whole fabric of NVMe devices, tens of thousands, hundreds of thousands of queues on tens of thousands of drives in a fabric, then you can treat NVMe like a cache line for DRAM. And you can be very intelligent about where you shard the writes, where you load balance the writes very, very intelligently. And at that point, you don’t need expensive SLC tiers to buffer writes. You’re not being sub-optimal, you’re not being unintelligent or treating the devices as blunt storage devices. You’re really digging into the devices and you’re making sure that you’re optimizing the actual underlying NAND flash well and cooperating with the controllers versus offloading work to the controllers.
Vik: So you don’t really see SLC as coming in on its own tier. Maybe it’s just going to be used as a buffer.
Val: Well, let me be, let me disclose here that I’m talking about how Weka has decided to optimize flash tiers. I think the industry needs SLC, right? The industry doesn’t have our patents, doesn’t have our implementation by and large, except for the server partners that we partner with. But generally, that technology is not available outside of Weka. So you do need SLC. There will be a rise, Nvidia is forecasting this themselves, even for CMX. There will be a rise in the need for SLC, QLC to buffer the writes when you can’t amortize them over TLC. But the upside of that is that if you buffer correctly to SLC, you can still use QLC. We’re skeptical, right? Because there again, these are not capacity workloads where you have the luxury of time to de-stage SLC to QLC. These are very bursty, very intensive workloads that are attempting to emulate memory. And so we think that ultimately one tier, whether it’s SLC or TLC is the only safe and fast way to offload KV offload to storage. But you know, the industry, there’s a lot of smart engineers across the industry and many vendors, the market will prove what works over time. We just know what works for us right now.
Vik: So in a NAND storage rack, you know, you could amortize writes over many drives. But do you also overprovision and to what ratio? Like if you have 100 drives in a rack and to make sure that you have the right endurance even after amortizing over TLC drives, do you actually have 20% more? What is the—
Val: These are tricks we can always play, right? So if we don’t have SLC and economically or just supply chain issues, we need to use TLC or God forbid today QLC only, then yes, you would really have to overprovision. And the net effect is you would be buying, let’s say a petabyte of storage but only use, only see 300 terabytes of effective or 500 terabytes of effective capacity from that petabyte that you’ve purchased and are powering because the overprovisioning is needed not to brick those drives.
Vik: Yeah, so it could be even a two or three is to one overprovisioning.
Val: Yeah, it’s tough, you know, because these are extreme workloads. There’s nothing gentle about emulating memory with NVMe with NAND flash, right? It’s a very intense workload.
Vik: Yeah, yeah, yeah. Well, that’s great insight. That’s really great insight for me on how SLC and TLC tiers work and what the tradeoffs are. So I definitely has added to my body of knowledge today.
Val: It’s nuanced. That’s why I said, you know, we could spend a lot of time on this alone.
Vik: Yeah, yeah, yeah. We should move on. I think a lot of interest in the market right now is for high bandwidth flash because now if people can put in flash right next to a GPU and have it store some stuff, it would be a useful thing, I would imagine. So my two questions I think around that are, does HBF use SLC, like single layer cells? That’s one thing. And secondly, what are really the use cases for this thing?
Val: Yeah, great question. I think HBF will probably have to use SLC in the most common configuration for the reasons we just discussed. It really is trying to be a great offload tier. It’s trying, in fact, in many cases to replace DRAM, right?
Vik: Right.
Val: And complement high bandwidth memory directly. For me, it’s always come down to a packaging decision. You know, I think SK Hynix has been public about the fact that they want to package it directly on the GPU package itself. They don’t want to go through buses, they don’t want to go through networks. They want it to be really tightly bound to GPUs. I think it’s a great vision. I haven’t seen any GPU vendor directly commit to that yet or adopt that yet, but I think in the future, maybe 2028, I think we’ll see that. But there’s other vendors that I can’t discuss on NDA that are packaging it differently on a PCI card, for example, and not even packaging it with GPUs, but with ASICs, non-GPU accelerators. So there’s going to be different packaging form factors, but ultimately, I think it’s a necessary thing.
Because the opportunity to get this right, the opportunity to optimize the KV transfer layer, the opportunity to schedule correctly. It was really acquisition, I think, of Qualcomm of Modular, right? And basically, yeah, and be able to get that compiler expertise. What Nvidia acquired with Groq was a lot of compiler expertise from the TPU team over at Google. That is going to be more important over time to making HBF really usable because as we recompile models to understand HBM and HBF tiers and maybe bypass, this is a prediction we have inside engineering, bypass DRAM tiers altogether because you can get a lot of bandwidth out of NVMe flash, NVMe, sorry, out of NAND flash devices over NVMe or other protocols or without NVMe, but just NAND flash, there are optimizations possible now and there’s definitely a trillion dollars of opportunity that encourages those optimizations between HBM and HBF with raw NAND flash and or NVMe.
Vik: You can always replace drives if you can run that bandwidth over a high-speed network and NVMe drives and like we spoke about overprovisioning and something dies, you change it out or whatever. HBF, I don’t know, the one thing that always is on my mind is it is NAND flash after all and it will, even if it’s SLC, it ultimately has an endurance to it. You can’t change it out if it’s packaged next to the ASIC or GPU.
Val: I agree.
Vik: So is that like something that’s going to play out or the lifetime of this thing is, I don’t know, 50 years, it doesn’t matter. We probably going to throw the chip away by then anyway.
Val: You’ve already done a good job in covering this. This is a bag of tricks engineers are throwing to mitigate this problem. It’s not one solution or one workaround. There’s overprovisioning is one of the tricks for sure. Then again, there’s proper amortization just at the storage at the NVMe layer, NVMe fabric layer. Then there’s very explicit scheduling at the inference at inference time, but also very explicit scheduling at model recompilation time and very intelligent token routing inside the MoE level and even just at the gross model level as well.
Vik: Okay, okay. So you could engineer the whole thing to a level that this is not really a problem and you can still use it.
Val: Not only can you do that, you know, I’ve already seen anecdotes of people applying Fable, right? To very complex engineering problems and seeing months worth of sophisticated systems engineering completed in four hours. So one optimistic scenario is that these agent harnesses and the eval loops, the quality eval loops and so forth, the judge judgment models and of course, the guard rail loops and the guard rail tokens, when you package that all together, some of these really deep tech engineering advances can happen much faster, but ironically, we need more engineering in the harness to make sure they happen reliably and safely.
CXL’s Future & AMD’s Mex
Vik: Okay. That’s amazing. Yeah, so much stuff is happening that it’s really interesting to see how all this plays out. So that’s the whole fun of this thing. And the one other thing that has recently cropped up is, and I know like we spoke about this last time and you had ideas around this that I want to see if it has evolved or changed ever since is the use of CXL. There’s a lot of talk about using CXL now because some of the RAM, DRAM isn’t really being used on every server. I think maybe like 50% is being used. So because memory is so expensive, even Google has had a change of heart, it looks like to actually start using CXL and reclaim some of the unused DRAM. What do you think like is CXL the thing now or?
Val: So CXL has a lot of fans and I was one of them 10 years ago. I’m not a fan anymore, right? And it’s not because I don’t like the technology, it’s because in the real world, there’s alternatives. And so, personally, I think Mellanox and of course, Nvidia’s acquisition and really great execution of scale up domains kind of killed CXL in one sense. The ability to have this great NVLink style network and have it used for memory, have it used for high bandwidth as well as regular DRAM memory, has been, has been, a real, glass ceiling or concrete ceiling really for CXL growth, market growth. There’s been less and less need. Then you’ve got Rocky, right? RDMA over Ethernet. And you never bet against Ethernet in this industry, right? So the challenge CXL has, it’s another bus. It’s another bus you have to engineer. It’s another bus you have to debug. It’s another bus you have to maintain and power. And it’s not that in isolation, it’s bad. And again, it’s got a great killer app today of utilizing, you know, this really precious underutilized resource in some cases of DRAM. But in the context of real world alternatives, I’m just pessimistic about the future of CXL. Largely because again, I’m biased. I’m able to leverage things like Rocky or NVLink over InfiniBand and deliver better than CXL performance with NAND flash economics, cost of goods and capacities. So it’s one thing to pull underutilized terabytes of DRAM. It’s another thing to pull petabytes and exabytes of NAND flash at the same or better performance.
Vik: Yeah, won’t a faster network bandwidth also benefit CXL like it does?
Val: Yes. The first principles are simple until you have to engineer the real world issues into it, but yeah, the first principles are, more bandwidth is good. And so if you can have faster than PCI bandwidth, to a pool of DRAM. As Google discovered, there’s benefits to that. It’s just that if there really were benefits to that, I think you’d have seen it in Blackwell, you’d have seen it in Vera Rubin from Nvidia or in Helios from AMD or even in Fineman, the next generation from Nvidia that’s already pre-announced, and you haven’t seen it, right? And then you haven’t seen it as a standard Supermicro offering, and you haven’t seen it as a standard Dell or HPE or Lenovo offering. And so I think it’s that absence that speaks volumes, right? There definitely are use cases for it, but for some reason it hasn’t broken out into mainstream.
Vik: Yeah, yeah, for yet, probably. It seems like, yeah, it seems like the shortage of HBM is now causing some players like Google to actually consider it now because it’s just like capacity issues are pushing hardware makers towards solutions they probably didn’t consider before. So, yeah, that’s another interesting thing to see how it’ll play out. Because you’re right, like, so all this time it hasn’t been in there. There’s a reason for that, right?
Val: There’s a reason for that. And I think there’s one important hint, right? So it was reported, I think by SemiAnalysis about two or three weeks ago that Nvidia changed the BOM on Vera Rubin. And they cut the actual amount of DRAM in half, either from two, four to two or three to one and a half, depending on the model. But that’s a clear indicator that A, DRAM has gotten too expensive, and B, Nvidia is projecting with CMX solutions in the marketplace from a number of vendors including Weka, that there will be less need for DRAM for inference. And so you’re seeing that, people are applying solutions to this problem and it’s not always just pooling it better, it’s just reducing it overall.
Vik: Yeah, yeah, that’s a part of what we spoke about in the beginning, right? Like, you could make optimizations to the algorithm, you could use sliding window, you could do a whole lot of different things, that, you know, maybe uses less memory overall going forward, rather than just use existing architectures but start pooling stuff together. So, yeah, you know, it could go either way. If we find better algorithms, then we probably don’t need to pull it. Could be.
Val: Yeah, yeah, that’s I think the prediction we’re making.
Vik: Yeah, speaking of better algorithms, what’s your take on AMD’s Mex acquisition? And for people who may not have heard of this, Mex is essentially a software company that found a way to optimize the use of DRAM by dynamically offloading all the unused parts of DRAM to NAND flash. And then, using AI to predict when that same information is going to be needed back in the DRAM and preemptively moving it back before the GPU even notices it’s gone, right? So, it looks like it’s a cool way to use NAND, but what’s your engineering interpretation of what’s going on?
Val: That was a fun one to review because I wasn’t familiar with Mex beforehand, but it was clear in reviewing their use cases pre-acquisition and the initial positioning post-acquisition is it’s AI technology, it’s machine learning, specifically small language models and small neural networks, in optimizing cache, caching algorithms and being more semantically aware than just least recently used or so forth, simplistic heuristics. So it’s the application of machine learning and deep learning into caching algorithms, but the actual use case is ironically enough not yet for KV cache offloading. It has potential to be very good there. We’ve seen even, things like popular in-memory databases, Redis and so forth, implement algorithms that aggressively de-stage DRAM to NAND flash and so forth and retrieve it back. And Redis does position that for KV cache offloading as well. So I think it’s an active space. But right now, I think, the AMD initially is targeting scientific computing. So whether it’s life sciences, whether it’s Monte Carlo simulations and other kinds of, seismic analysis, weather analysis, those are the kinds of applications that that technology has proven itself in and we may see it happen, we may see it appear in KV cache offloading as well.
Vik: Nice. Yeah. I thought it was an interesting use of the predictive nature of an LLM because, if you can predict what the next word is, why not use it to predict what the next page of memory is required and quickly pull it from flash to DRAM. I don’t know how real practical or useful it is, but I found the idea was interesting.
The AI Flywheel & Edge Inference
Val: I think your instinct is right, you know, why is Cursor customized, Kimi K 2.5, it’s a composer. Many companies are realizing now that the bar towards being able to train your own model has come down. It’s a much more accessible thing for many companies right now. You don’t need million dollar ML researchers to train your own models anymore. And when you can customize a model for your domain, it actually doesn’t have to be an LLM at all. It can be an ML, it can just be a very tight neural network. It can be inference on a CPU. It can be inference on a small low power CPU if the model is really domain specific. And it’s just a neural network at that point. It’s not a large language model. And there’s I think going to be again another Cambrian explosion of use cases and applications for clever small models that do things that heuristics, peak that and can no longer optimize or improve.
Vik: Yeah, do you think that this use case is basically physical AI and robotics, where you could have those sensors at the edge, like process very specific amounts of information. It’s only one kind of information from a sensor, right? So it’s not like a large model you need. So is that a useful use case you think going forward?
Val: 100%. I think it’s probably going to be the reference architecture for robotics and edge inference. We don’t need large language models. There will be some aggressive cloud connection, whether it’s through Starlink in remote locations or just broadband, if you really have to, burst to some kind of complex decision that a large language model has to make, but 90, 95, maybe 99% of inference for robotics will be local and disconnected, air gapped, so to speak.
Vik: Yeah, that’s the fact that you could make inference decisions at the point of sensing, or, just like put an intelligence anywhere, actually, is a very useful edge use case. How useful or how good that intelligence is, I think yet to be seen. But in principle, you could deploy these little models everywhere.
Val: Yeah, the cost of these Raspberry Pi style system on a chip motherboards are really plunging. And fortunately, again, whether it’s KV cache optimizations or just, small model quantization and just custom neural network training that doesn’t have to be a large language model at all, is intersecting really well with really affordable, system on chips, SOCs. And yes, that results in some really interesting robotics and drone use cases, yeah.
Vik: Yeah, didn’t Jensen mention something about the AI flywheel in this context?
Val: Exactly. So this is a general concept of, it’s all about being able to capture data, domain specific data, train models, and then customize that with more either real world domain specific data or synthetic data now that you have enough real data to create useful synthetic data and just keep iterating on that loop of you’re pushing the frontier with big models, you’re customizing either through fine tuning, distilling, quantizing, low rank adapting, etcetera, all sorts of customizations. You’re customizing smaller and smaller versions of those models. You’re able to maybe retrain entire small neural networks that are very domain specific of those models. And just get more and more efficient at processing inference, retaining some of the new fresh data and keeping the flywheel going. So it’s a mix. It’s definitely a whole ecosystem. It’s a thriving ecosystem of different model types, different phases of data, different types of data. But if you keep the flywheel going, it stays relevant with nature’s natural entropy. So yes.
The Tokenomics of Software
Vik: Yeah, that’s a fascinating idea. Since we’re coming up on time, I want to pick your brain for one prediction. What do you think is going to happen in the next 12 months? What are you most excited about?
Val: So zooming out a bit, again, this is something Jensen references very, very often is that software is fundamentally changing. A year ago, we wouldn’t have predicted that all of our engineers really would be using AI for most of their daily work right now. It was heresy even a year ago. And so the not only the rate of change that’s happening right now, but the fundamental change in software is that more and more software now is not compiled and run. It’s basically compiled, run and inference, right? It’s really agents now, as we said, being much more intelligent in their token consumption. We’re not using Opus for everything. We’re not using GPT 555 for everything. We’re definitely using now model routing and a mixture of models and a mixture of experts within models right now to be very token efficient. And what that means is now the cost of running a software business is radically different than before.
It is a high marginal cost. You can’t just leverage, even with, maybe KV cache is the way to leverage, but you can’t just leverage the cost of tokens across users the way you could leverage cloud instances and databases and VMs and micro VMs and containers across users. And SaaS companies, the reason I believe the SaaS apocalypse is real and is a problem is that no matter how much SaaS companies figure out new pricing models and new values, value-based pricing and so forth, their OPEX is going through the roof. Their OPEX now is token OPEX, it’s tokenomics, it’s token consumption. And yes, all of these engineering solutions we just discussed are ways to manage those. But if you just take a look at token volumes on OpenRouter, week after week after week, it keeps rising and rising. And we really just barely begun mainstream token consumption and persistent agent swarms. The only way to run a profitable gross margin business in software will be to own more of the token stack. And you can continue to outsource that to an inference provider, to a model provider, or you can acquire it. You can merge with a neocloud, you can merge with a token factory and you can essentially, vertically integrate more of that very expensive token generation stack and continue to run the high gross margin software business.
Vik: Oh, that’s fascinating.
Val: So the real question is, with cash flows what they are, will SaaS giants acquire neoclouds before neoclouds are able to acquire the SaaS giants?
Vik: That’s fascinating. You know, I always spoke about, the best way for large companies to save on token costs is you bring inference on premises, right?
Val: Exactly.
Vik: So you could run that at the edge. What you’re suggesting is one level, that concept on steroids. A big enough software company can go acquire a neocloud and say that this is my token factory. And now I can, there’s still a cost to run the token factory, but the tokens are yours to use. It’s entirely yours. Now, if every software, big software company starts acquiring neoclouds of some size, I mean, they don’t have to be multi-gigawatt data centers, but—
Val: I was predicting, I made this prediction before xAI acquired Cursor. I predicted, like Workday or monday.com or something like that would merge with a mid-tier neocloud, but now, of course, first domino’s fallen with xAI and Cursor. And older established SaaS companies are going to have to react as well. So yes, I think whether the dominoes start falling in the middle or one side or another, it’s kind of inevitable now that most SaaS companies and most neoclouds will have to merge.
Vik: That’s a fascinating prediction. I would love to see how that works out. Yeah. Thanks so much, Val, it’s always a pleasure chatting with you. You’re a fire hose of information that I know I’m going to listen to this podcast later myself as I’m reviewing the edits and stuff and be like, oh my God, I missed that when I spoke to Val. But, I hope this helps all our viewers as well. It’s really a pleasure.
Val: Always a pleasure. We said we’d enjoy it next time last time we did it and I’m definitely looking forward a few months from now from coming back. We should come back but well before the end of the year because by the end of the year again, we’re going to be very surprised by what happens.
Vik: It’s an eternity. Every three months is an eternity in AI time. So yeah, we should do this more often.
Val: 100%.
Vik: All right, guys, that’s it for today. Thanks for listening. If you’re enjoying Semi Doped, please share it with your friends. And we also have a daily newsletter on semidoped.com where we put our daily takes on the news. It helps us keep abreast of what is happening in this fast-paced AI landscape we are in. It’s entirely free, so make sure to check it out. And thanks for everyone who puts comments on YouTube. We do read all of them. Some of them are really amazing, some of them are really funny. We have a good laugh. But we read all the comments even if we don’t respond, we promise. And it helps us plan all the future episodes, so definitely keep them coming. And if you can, leave us a five-star review on Apple podcast, that really helps us out. All right, cheers and catch you on the next one.
