Austin talks with Saurabh Dighe, Corporate Vice President of Azure Hardware Systems at Microsoft, about the new Maia 300 AI accelerator. Saurabh explains how the system is co-designed from the silicon up to the data center to tackle emerging agentic AI workloads by focusing on data movement efficiency. He details the architecture, including its 96-accelerator rack design, 18 TB/s of HBM4 bandwidth, and a scale-up fabric connecting over 12,000 chips.
This episode is presented by Crusoe.
Crusoe’s Serverless Fine-Tuning trains an open model like Llama, DeepSeek, or GLM on your own data, and you pay per million tokens, not per GPU hour.
About $2 buys roughly 5M training tokens on smaller models.
Get $5 in credit to start: https://semidoped.com/crusoe
Things we cover:
Maia 300’s 7x performance uplift
Designing for agentic AI workloads
HBM4, chiplets, and the N2P process
A two-tiered scale-up fabric for 12,000+ accelerators
Rack-scale efficiency and cooling
Using agents to optimize software kernels
This podcast is lightly edited for clarity.
Microsoft’s Silicon Strategy
Hello listeners and welcome to another episode of Semi Doped. I’m Austin Lyons with Chipstrat and with me today is a special guest, Saurabh Dighe, Corporate Vice President of Azure Hardware Systems and an architect at Microsoft. Welcome, Saurabh.
Saurabh: Thank you, Austin, for having me over. I’ve really enjoyed your YouTube videos, so I’m glad that we’re doing this one as well.
Awesome, I appreciate that. For newer listeners, you may not know, but Saurabh and I talked back in January about the Maia 200 from Microsoft. Today he’s back 10 months later, and we’re talking about Maia 300. So, with that, remind us at a very high level, maybe for new listeners, why did Microsoft make Maia 100 and 200, and what are we talking about today with the 300?
Saurabh: Absolutely. It was a great conversation in January, so here we are 10 months later. Before I go deep into Maia itself, which I’m sure we’ll do, it’s important to explain why Microsoft is building custom silicon and systems. We have this privilege of running a planet-scale infrastructure, and we run a lot of workloads—SaaS, PaaS, IaaS—businesses where we are able to understand our customers’ needs and requirements, and really to understand what infrastructure requirements they need. We’re not here to just build a chip for a chip’s sake; we’re here to really differentiate and deliver value to our customers.
That philosophy for us is not a chip story. It’s about how the whole system works together to deliver that value. We have been on a multi-year silicon and systems journey inside Microsoft. You may have heard about our compute silicon, which is Cobalt. We’re doing Maia, of course, for our AI. We do Azure Boost for our front-end networking. We do Azure DPU for storage processing and data processing. And then we have a lot of custom security silicon chips. We have been embracing this philosophy of building our infrastructure that delivers value to our customers and gives us the freedom to innovate so that infrastructure can then become something that the customers can rely on at scale.
The Maia journey really started for us with Maia 100, as you mentioned, but this was before the world of LLMs. We were starting to look at what it takes to accelerate speech and convolutional neural networks, because that was the growing trend at that time. Then we quickly entered the regime of LLMs when we decided to build Maia 200 for how to build the best AI infrastructure for LLM inferencing. This was a bet we took because at that time, people were talking about training large models. We looked ahead and said, even though training is going to be super important, the distribution of that intelligence is going to be equally important, if not more. This is where we took an inference-first approach for our Maia system. Again, it’s not just a chip story, but it’s really about how we can build the best AI infrastructure system and take the software, the orchestration that goes around it, the model families that need to be ported on, the custom silicon itself, the memory subsystem, the networking and data center, and wrap the whole thing into how we maximize the token throughput and the intelligence we can deliver for every watt or every dollar that we deploy.
So that’s our philosophy behind the Maia architecture. We disclosed Maia 200 earlier in the year. Today we are unveiling the Maia 300 architecture. With the Maia 300 architecture, we will talk about the foundations that we built on, but then how we developed a generational leap in delivering performance per dollar and performance per watt with this new system and architecture.
Agentic AI and Workload Shifts
This is all great framing. I love that you started with planet-scale because it’s easy to forget just how massive Microsoft Azure’s reach is. Therefore, especially when it comes to the defining workload of our time, LLM inference, how big the opportunity is to co-optimize and co-design the workload from the software to the serving layer, all the way down to the infrastructure and the custom silicon. And to your point, not only the accelerator, but the memory decisions and the interconnect decisions, as I know we’ll get into scale-up here in a bit.
I can definitely see how this is a huge opportunity to co-design. We’ll talk about workloads too here in a little bit, but just to remind listeners, even just first-party workloads for Microsoft like GitHub Copilot, Microsoft Copilot, and Microsoft Foundry—you guys are already deploying inference at scale, and that’s not even counting third-party workloads. But maybe talk really quick at a high level about the workloads, and then we’ll get into the architectural differences with the Maia 300.
Saurabh: Absolutely. Since we talked in January, the shape of the workload and the computational profile has definitely changed. We are moving from a predominantly chat-based experience to now agents with long-running tasks. That has definitely made us rethink how we build and deliver infrastructure for this new agentic computational profile.
We are starting to see that the number of tokens every agent generates is a lot more than what humans can generate. We’re starting to see more persistent KV because the context is getting longer. We’re able to see that the agent is planning, then doing a tool call, then ingesting new documents, then running an inference and iterating that loop. So KV is getting more persistent.
We’re also starting to see how important it is for memory bandwidth and capacity to be a first-order design principle. We’re also starting to see how important it is to pack more and more efficient compute in a given power and capital envelope, or energy and capital constraint. All of these changing trends and factors, we’re able to see firsthand with the workloads deployed at scale, how customers are using it, and being able to get that telemetry out of the systems. At the same time, we design the entirety of the system to then maximize the intelligence you can deliver in that constraint. So yes, workloads are definitely changing, but with Maia 300, we feel very confident that some of the decisions we made allow us to extract the right cost point, the right latency point, and the right performance—think of it as the right class of service for that particular workload. And like we talked about, we tend to talk about workloads, but they’re not monolithic. As we talked about last time, there is this frontier efficiency curve, and we do believe that one size does not fit all. When we talk about Maia and some of the architectural decisions we made, we made sure the system was balanced, but it was also leaning towards where we believe longer context, memory-bound, long-running workloads are. That’s where I think we can extract the maximum value out of the system. And there are other systems that we will deploy; we will deploy a heterogeneous fleet to make sure that we cover every point on that frontier efficiency curve.
Maia 300 Architecture: Data Flow and Low Precision
Yeah, that’s good. And feel free as we talk through these architectural decisions, like you mentioned, the workload now demands with the agentic, like a long-running kind of persistent KV cache, lots of memory bandwidth, memory capacity, but continue to sprinkle in your thoughts on how the workloads are shaping your design decisions because when we see that Pareto frontier with throughput on the Y-axis and interactivity on the X-axis, I’m always interested in what the actual deployment and the companies are choosing. Obviously they’re not necessarily filling out that whole curve, so I’m always curious what is the class of workload, like you said, that would land at a certain point? Is it higher throughput or lower latency? So again, as we’ll go through this here, but definitely feel free to sprinkle that in because I don’t think we get a lot of detail about exactly which points people are sort of optimizing for.
Saurabh: We’ll do. One of the things that we believe in as we look at the first principles of how you’re designing an AI inference system—AI inferencing to me is not a compute problem only. It’s actually a data movement problem. And so we decided that instead of trying to attack the compute problem, let’s see how we can create a new class of architecture that really focuses on the data movement energy. The industry has done an excellent job over the last few years with new microarchitecture, new process technology, and new… but what has not kept up is reducing data movement, because that is a significant factor—sometimes two orders of magnitude in terms of the energy you could consume to move data and keep the compute engine fed.
One of the fundamental principles of our architecture is what we call the software-defined data flow architecture, which we introduced in Maia 200 and continue to enhance. The underlying thesis is that with AI inferencing, we can actually plan for the data movement ahead of time. Because we can plan for the data movement ahead of time and we know what these data movement patterns look like, we can have distributed memories and pin the data at the right location for the right time. We can hide communication and computation and continue to improve the efficiency of data movement in this architecture.
Now with AI-assisted kernel development, it has become even more efficient. We don’t need armies of software people to program this system; we can have agents optimize kernels. To your point, if the workload is LLM inferencing, at the heart of it, we believe the data flow architecture gives the best energy efficiency for inferencing per token. So that’s where we started.
At the same time, we said, okay, that’s great, but we need to be able to optimize the compute engine and the data engines together in one system. We then took an approach where we had distributed memories on the die, so we use a lot of SRAM. We use it as a scratchpad, and we expose the programmability and the explicit data movement instructions back into the compiler of the software layer. We’re also able to create much better computation and communication patterns on-die. We have the SRAM with our tensor unit and our vector unit, but we also have a global NoC that distributes the data within the chips inside the package. That really helps us again reduce the data movement.
We’ve gone towards a more NUMA-based architecture, which means that once you’ve placed the data close to your compute, you don’t have to drag it from another memory controller or PHY sitting on the other side of the chip. A lot of thought has gone into how to keep the data movement, data storage, and data computation at the lowest energy possible.
As we move outside the SoC, we look at how to connect these chips together. A lot of times people think of this as just putting a network to connect these chips together, but we thought of the network as an extension of the data flow architecture. What that means is that specialized data movement engines, or DMAs, and the communication protocol that goes on top can allow chips in other nodes, servers, and racks to talk through the same communication paradigm as what they’re doing on the chip as well. It’s just an extension of that data flow architecture.
Finally, we also made sure that we drive a lot of rack-scale efficiency. Because in the end, what matters is the deployment. People don’t care about just the chip; they care about how the entire deployment delivers more tokens at a lower operating cost. That’s where our rack-scale architecture really helps us build a more efficient deployment scenario in our data centers. So, there are lots of layers of optimization, right from the silicon itself all the way to how we deploy and allocate power in the data centers.
This episode of Semi Doped is presented by Crusoe. They build the power, the data centers and the cloud behind AI. Did you know Crusoe can help you tune an open source model on your own data for just a few bucks? Frontier models are very good, but they don’t know your company’s terminology, your part numbers, or how you like your reports formatted.
So, you keep stuffing instructions and context into the prompt of an expensive frontier model to get the job done. But for simple repeatable work like reports, an open source model trained on your data can actually match a much bigger model. Think of a new grad with three months of on the job training. You don’t always need the PhD, and the open model is cheaper and faster to run.
Fine tuning used to mean renting GPUs, working on orchestration and Kubernetes, writing training code and babysitting the entire setup process. But Crusoe’s serverless fine tuning removes all of that. Upload your data sets, pick an open model like GLM, Lama or Deep Seek, and they handle the training. You pay per million token used, not per GPU hour.
And on the smaller models like QN 38B, $2 can take you a long way with millions of training tokens available. Once the training is done, you can then deploy them on Crusoe Cloud in just one click or download the weights and run them anywhere. Your data only fine tunes your model. And it doesn’t have to be work data either.
You can try it on how you write emails. Pick a task, compare the tuned model to your current prompt and see how it does. Crusoe’s giving all new accounts $5 in credit to get started today. That’s enough to start tinkering with fine tuning right now. Links in the description.
Maia 300 System Specs: Racks, HBM4, and Chiplets
Okay, so the framing with the Maia 300 and agentic AI is all about minimizing data movement. You’re co-designing all the way down from the amount of SRAM on the chip to the local HBM, to the software for developers to manage this, all the way to the networking and how you’re communicating across chips because, of course, on a scale-up fabric they’re sharing memory, so to speak, with DMA. All the way to rack-scale, how many accelerators can we fit into this rack, and so on.
For listeners who haven’t read the recent announcements, can you just zoom out and give us the high level of what the Maia 300 looks like? How many chips are in a rack, and so on? Then we’ll dive into the memory specifically. I want to dig into the SRAM versus HBM tradeoffs and decisions you made and also maybe comparing against Maia 200, like HBM3E to HBM4. But yeah, maybe just zoom us back out and remind us at a high level of some of the specs for the Maia 300.
Saurabh: Absolutely. What we’ve announced today is really our conviction of the Maia inference system architecture. We have Maia 200 in production, outperforming some of the GPUs that are in our fleet today. With that evidence and production-proven success, we are saying that we’re going to be scaling Maia to multiple gigawatt deployments starting in 2027. That’s a pretty significant strategic positioning for us as Microsoft, and it’ll be a foundational element of all Microsoft products and services in the future.
Now, Maia 300 specifically, what we’re saying is that over Maia 200, we are expecting this rack-scale architecture to deliver 7x more token throughput per rack than what we could do with Maia 200. And it will be delivered at 2.5x more performance per watt.
So, the interesting story now is how do we get there? That’s what we’re excited to talk about. As you look at it, we talked a little bit about the software-defined data flow architecture—that’s really a foundational element. We also went to multiple chiplets. We went to a TSMC N2P process, and with a chiplet architecture, we’re really able to balance how much flops we want, how much HBM we want in the system, and how much IO or scale-up bandwidth we want in the system. So we’ve tuned the system a lot more to delivering, again, not stranding flops, not stranding resources in the system, but delivering that useful intelligence through every watt. A multi-chiplet architecture really helped us there.
In addition, we’ve introduced much better arithmetic, so low-precision arithmetic. We can talk about how with agentic use cases and with more persistent and growing KV, these MXFP4-like formats really help us optimize our memory. With lower-precision data formats, we have optimized model weight storage in four-bit or lower precision. At the same time, we can optimize the KV with some of these lower precisions. On the compute side, we can have computation that’s FP4 by FP8 so that we can continue to keep the accuracy and the quality of the model output, while at the same time packing more and more concurrency and higher KV in the same HBM. So, a lot of thought went into how we use lower-precision math.
A lot of thought went into how we actually design the memory subsystem. Like we said, we have the on-die SRAM, but at the same time, we’ve also gone to HBM4. We have got eight stacks of HBM4 on Maia 300, delivering a total of 18 terabytes per second of HBM bandwidth. That’s almost 2.5x of what we did on Maia 200. And if you look at the rack-scale level, that’s 5x of what we could do on Maia 200. That really was a focus for us again, going back to how do we manage these longer contexts, more persistent KVs, and decode-bound or memory-bound, decode-heavy workloads.
Our scale-up network is the same scale-up network that we use from a server all the way to a superpod, which is now 12,288 accelerators in a superpod configuration, or 96 accelerators in a pod, which we call a rack. In a rack, we will now have 96 accelerators, which is double what we did on Maia 200. And even though we doubled the number of accelerators, we can deliver this at 167 kilowatts per rack, which is extremely energy efficient if you compare it to anything else that you’d be getting from other vendors next year. So, 96 accelerators at 167 kilowatts and then being able to scale up with one scale-up fabric from rack to superpod to about more than 12,000 accelerators is really how we’ve spent our time creating a really unique scale-up fabric.
Fascinating. Let’s get into the scale-up fabric here in a second because it’s very interesting to go so hard into the scale-up fabric and make that a big bet there. So I want to unpack that. But first, the HBM. You went from six HBM3E stacks to eight HBM4 stacks and the bandwidth went up, I think you said maybe two, two and a half times, and capacity went up per accelerator. So zooming out, the amount of bandwidth and capacity went up more than the compute. So maybe again, it sounds like you were really making decisions here around the decode side of things and the memory side of things for KV cache. How do you decide—obviously if you add more bandwidth and capacity, you can actually unlock more of the flops if you’re not keeping the compute waiting on the memory. How do you thread that needle and decide how much more of your die area you want to commit to flops versus HBM? And also, relatedly, remind me, do you guys have just one compute die with the eight HBM stacks around it? And then maybe contrast with others who are doing two compute dies.
Saurabh: This is a great question and it really comes down to what problem are we trying to solve and why. Like I said, we continue to focus on inferencing as our primary. For inference-based workloads, we wanted to make sure that we had a balanced system where we are able to expand our memory bandwidth and capacity, like we talked about, while keeping flops continuing to increase. We did move to N2P because we believe that that’s the most efficient process node for computation flops. We went to a full-reticle size compute, but at the same time, we felt that going towards two compute dies would have not been the right architectural choice for the workloads and the shape of the workloads that we are seeing.
Instead, what we did was we still wanted the eight HBM stacks on HBM4. This is where our chiplet architecture really came into play, where we were able to create these memory interface dies to extend the shoreline on either side of the die. We’re able to extend the shoreline with a chiplet architecture, put four HBMs on either side of that, while still keeping it as one very efficient compute die.
Now going from HBM3E to HBM4, you get those two multiplier effects. On one end, picojoule per bit is super critical for HBM data transfer. With HBM3 to HBM4, the industry has moved to a wider interface but kept the pin speeds relatively the same. What happens is you end up with a much better data transfer rate and energy profile for the data transfer rate. That’s one multiplier you get. And then obviously the second multiplier is when you go from six stacks of HBM3E on Maia 200 to eight stacks of HBM4. So if you put those two things together along with our multi-chiplet architecture, you go from something that’s delivering 7 terabytes per second with a certain amount of flops to now two and a half times that—18 terabytes per second—and with another around 35% to 37% of flop increase from something like 750 watts to 950 watts. With just a 200-watt increase, we were extremely cognizant of the fact that when we put power into these SoCs and into our systems, it needs to translate to significantly better intelligence outcome. We did not want to strand any of those resources. So those are the thoughts that went behind balancing the machine towards these workloads.
That’s an interesting figure. Got it, got it. Yes, increasing your bandwidth per watt is a very interesting figure of merit and it makes a lot of sense and it aligns with your earlier framing of thinking a lot about data movement. Okay, so you’re talking about a lot of decisions that are very beneficial for decode, for the memory-bound part of the workload. Maybe talk to me a little bit about pre-fill. I mean, within a rack of 96 accelerators, will you still ultimately disaggregate pre-fill and decode and what does that look like?
Saurabh: Just a quick correction. We go from HBM3E to HBM4, not 4E. But everything else you said, absolutely correct. It’s really two things, right? One is we are trying to optimize both for energy efficiency and cost. When you look at energy efficiency, you get a much better picojoule per bit, or overall transfer bandwidth per watt is a better profile when we go to the wider interface; you get a lot more bits coming out. At the same time, we also wanted to make sure that we are using the right process technology for the right function. For compute, you want to get to N2P. For some of the HBM and HBM PHYs, we decided that we’ll have a different process technology to be able to manage the cost and the power profile of going to HBM4 as well. That really drives our philosophy on how we make sure that we are more cost-efficient over a generation and more energy-efficient per generation. The decisions to go chiplet really helped us there.
Absolutely. We did a lot of modeling on understanding what our pre-fill to decode ratios are inside the fleet. We’re pretty confident that even with what we have in our chips and over the rack, we can absolutely deliver the pre-fill to decode ratios. You bring up a good point: what is the pre-fill decode disaggregation strategy? This is where our strength in networking comes along because we have one scale-up fabric. We can absolutely take instances of pre-fill that have multiple GPUs and position them in one rack or spread them across racks, and then instances of decode in the same scale-up fabric. This really helps us because we don’t have to go over a scale-out network to pass the KV from a pre-fill to a decode position. We can do that all over the same scale-up. We do run our fleet with disaggregation and we are able to take some of those Maias and run them as a pre-fill instance and some of them as decode. When we match-rate them, depending on how many pre-fill machines you have and what is the token output on those machines, we try to match that with our decode side as well.
Scale-Up Fabric and Network Topology
Nice, got it. Okay, so let’s go into networking topology. I know yours is two-tiered, but tell us, zooming out, people like to think about where is it copper, where is it optics? How fast are the links? How many accelerators does it connect? And maybe even at a high level, if you can fit 96 accelerators in one rack and you’re doing mixture-of-experts inference at scale, why do you even need to go beyond one rack? Why need superpods?
Saurabh: It’s a good question. Let me build it up from the bottom. At the lowest level of our network topology, we have our integrated NICs. We introduced them in Maia 200 and we continue to advance the capability of those NICs. They are on-die NICs, extremely area and power efficient, and they’re really built for scale-up. We’re not trying to solve for millions of endpoints talking or a general cloud network NIC kind of thing. We’re basically looking at what a scale-up domain looks like. It needs to be extremely high performance, it needs to be low latency, and we are going to continue to increase the bandwidth. That’s what a tensor parallelism or some form of workload that’s spread across nodes in the rack would need. Looking at the communication patterns, we’ve optimized the NIC.
From there, we have used standard Ethernet switches and cables because we do believe, again with the philosophy of maintaining the cost and not trying to go into more exotic implementations, we’ve stayed on Ethernet to really benefit from the scale economics of that industry. We have built a node topology that keeps some of those links within the server itself because we believe that neighboring Maias would have much higher communication, especially if you’re doing tensor parallelism. So we’re taking the tensors and spreading them across a couple of Maias in the node. A lot of the bandwidth stays within the node and then half of that bandwidth goes to the switch links. That’s the first tier of switch, and all 96 accelerators have connections to the first tier of switch.
Then we also decided that as models grew bigger or as we started to look at higher sharding schemes, or we thought of flexible job placement so that the software allocator’s orchestration becomes a lot easier, we felt that a second tier of network would really help us scale into those use cases. We have our own transport on top of Ethernet, so we were able to then use some of those transport features to be able to say, yes, I can actually scale now and have a second tier. We can have accelerators from one rack talk to another accelerator from another rack using the same communication stack that’s been developed for in-rack as well as cross-rack. That really helps us now get into this place where you can start expanding the cluster size to about, let’s say, 128 racks in the same scale-up domain.
Copper has a reach, so we stayed with copper inside the rack. But as soon as we went from the switch in the rack to the next level of switch, we started using optics. You’ll start seeing some of those systems come next year as well, where you’ll start seeing more and more optics being used in the scale-up domain, really to find that reach and to build these larger and larger pods or superpods.
Got it. Okay. This is really interesting. So you said at first, you don’t really need to design around scale-out because you’re not trying to do a million accelerators for a training job; you’re just focused on inference as we talked about before. So you’re really focused on scale-up. You said there are four accelerators that are tightly coupled to each other, directly connected, and you’re thinking about splitting up the workload so that neighbors talk to each other a lot directly over copper. All the accelerators in the rack are connected via the first level of switch to each other, also copper. And then the switches connect to other switches, rack to rack, using optics. Awesome, interesting. And it sounds very powerful and flexible. Do you have any two cents or are you guys saying anything about how Microsoft feels on near-package optics versus CPO, NPO versus CPO?
Saurabh: The story is developing. We do believe that optics in scale-up is something that everybody will eventually move towards. At the same time, I think we need to bring the reliability and the cost of optics along with the right supply chain motion to make sure that the industry can all move together in some ways. In terms of NPO and CPO, there are real trade-offs there. Talking about what we do with reliability, with NPO, you have a less disruptive, much more streamlined path into the server and how you design it. But over time, we also know that CPO gives you the power efficiency at extremely high bandwidths. I think the industry is going to start with a little bit of NPO, trying to get a feel of how we manufacture this, how we make this very serviceable, how we improve the reliability and the cost, and then move towards a more integrated approach, which is CPO on the XPU side. Similarly on the switch side, we’re already starting to see some of those motions as well.
Rack-Scale Design: Power and Cooling
Yeah, makes a lot of sense. So then talk to me at the rack scale. Let’s talk about power and cooling. I saw a sweet picture with this big thing sitting next to a rack of the 96 accelerators. What was going on there? And when you’re talking about cooling, are you thinking about deploying this only in places that already support liquid cooling or can you go into air-cooled data centers? Why did you make these cooling decisions?
Saurabh: This is again where the co-design aspect of where we play out really well. We have designed Maia to be deployed in both an air-cooled and a liquid-cooled data center. We know that as we expand this to global and planet-scale, there are going to be opportunities to say, hey, there’s a data center here and it needs AI infrastructure, and sometimes you may not have access to liquid-cooled facilities because they are still being developed.
This is a great, opportunistic deployment for us to say, hey, we can actually pair a Maia with a heat exchanger unit. Again, going to the power of the energy efficiency of the rack at 167 kilowatts, we’re able to actually design what we call a heat exchanger unit. What it is, is it takes the Maia chip itself, which is liquid-cooled along with other components inside the server, but then when the warm water comes in, it can be put next to a rack that has fans.
Sure. Yeah, that’s really cool. I like that you keep the optionality to be able to deploy these wherever you want and if it can’t support liquid cooling normally, then you have this heat exchange unit to make it happen. So, but that makes me wonder then, how do you guys decide where to deploy these? Are you putting them in the same place where you’ve got the Maia 200s? Are you trying to put them for inference near densely populated places to have the inferences as close as possible? How are these decisions even made?
Saurabh: Well, that is definitely a complex equation. But going back to starting with our customers, we are seeing them use a lot of our Microsoft products and services. We talk to our internal product teams and services and understand where the demand is, where the growth is, and where capacity needs to land. Then based on the demand and growth, we have a pretty well-oiled process inside the company to make sure that the systems are ready for those data centers.
At the same time, we want to make sure that these systems are delivering that class of service that is wanted. We do a lot of work internally to make sure that the services are first, in some ways, running shadow traffic on it just to make sure everything is correct and hitting the right customer experiences and expectations. When the services are ready to scale, we have them deployed in the regions that they want.
Software, Deployment, and The Big Picture
That makes sense. Okay, so when you’re talking about your internal customers, how much are they using Maia 200 today? And then as far as Maia 300 goes, from a software perspective, you talked about the software-defined data flow architecture. Is it going to be a big lift for those internal customers to move to the next version of Maia?
Saurabh: No, absolutely not. In fact, this is really at the kernel level where all these optimizations play in. We have a very robust programming stack that’s already ready, so we can build the right kernels. But what I’m really excited about and what has changed since maybe the last time we talked is the ability to actually program these machines really well and extract almost peak performance out of this machine. That’s because of how we can now take agents in the loop and use agentic kernel optimization to build and understand and put the right optimizations in place even before the customer or the application sees it.
For them, this is mostly an API endpoint kind of a thing where they don’t have to worry about the underlying architecture. That’s abstracted away from them. All of this data flow architecture, the placement of the data, how we continue to increase the performance of those specific kernels—that’s all hidden from them. That’s part of the library that gets built along with these systems.
Gotcha. So it’s not a problem for your customers and as far as internally needing to upgrade the kernels—because those specific kernels are just the things running over and over and doing all the math—you’re saying that agentic AI coding capabilities have progressed so quickly in the last six to nine months that you can also throw a swarm of agents at it and that helps you even more quickly optimize these kernels.
Saurabh: Exactly. Day-zero model availability is not going to be a problem. You’re going to be able to really farm these out. If there are new models or new kernels that need to be written, these agents are able to continue to run in a loop and create new versions of optimized kernels that we can then populate back into our kernel libraries that can then be exposed to just LLM calls.
That’s amazing. So, as you stand these up to serve internal customers, essentially tokens-as-a-service, do you ultimately configure them differently to meet different SLAs? Like does someone want five milliseconds of latency, someone else is okay with 20 milliseconds, but they want more throughput or cheaper tokens? How are these getting configured?
Saurabh: The answer honestly really depends. Like we said, and you mentioned, there’s an interactive versus throughput chart that people look at. When you actually look at the charts, they are different points of sharding and different points of optimization of the same system. Depending on where your workload is leaning towards, if you have chat or reasoning heavy, you will have a certain aspect of what the workload is stressing. Then you also want to understand what your interactivity is. If it’s, “Hey, I just finished a Teams meeting and I just want to summarize what it is,” well, fine, we could just batch that off and make it a throughput equation. Whereas if this is a super-critical agentic loop with extremely low latency requirements, yes, you have to reduce the concurrency and batching of what you can do in that system so that most of the resources are given to that particular task.
The systems are flexible to some extent. Now, there is model-infrastructure co-design that can definitely happen. For example, when we work with our own model teams, we can expose the strengths of the system to the model team. What they do is now they say, “Oh, great, now I know what this system is capable of,” and some of the model infra decisions can be made in a better way. That can also help continue to extract more and more out of the system. But I would say there are those two layers of optimization you can do: how much are you really looking at model-infra co-design, and then how are you looking at the task that’s coming to you and finding the right optimal configuration, sharding scheme, and interactivity point? Or again, going back to what is that class of service and capability versus price that I have to deliver for this particular request and how do I do that in these systems? You can make that optimization decision at runtime, think of it that way.
Sure, sure. Okay, so zooming way out, last question. We’ve covered so much ground here. This has been awesome. How should we think when we look at Microsoft and the Maia 300, Microsoft Azure, how should we ultimately think about the benefits to Azure? Is it that Azure can now run internal workloads more cost effectively? Is it, oh, here’s a vector where actually maybe there’s an opportunity to rent these out or sell them to other customers? How should we think about the big picture?
Saurabh: The big picture is how our customers really benefit from the investments we are making. I think that’s the real story here. It’s not about the chip, it’s not about Maia itself, but building an AI inference system that really maximizes the token throughput or the intelligence you can create out of a given power envelope or a given cost envelope or even a capacity constraint.
Yeah. Definitely. Okay, so to summarize, it sounds like you’re saying, hey, the world has a fixed amount of power and a fixed amount of dollars and we’re just trying to help customers get more intelligence out of their budget, out of their power budget, out of their financial budget, and the Maia 300 is a step in the right direction, co-designed with thinking about minimizing data movement and co-designed from accelerator to memory, trade-off decisions to network, to cooling and racks, all the way co-designed to just serve the customer and try to do more for them, essentially.
Saurabh: Exactly.
All right. Well, I’ll be a Microsoft salesperson for you. There you go.
Saurabh: Thank you, Austin. It was such a pleasure.
Okay, yes, always a pleasure and, you know, let me know when you’re ready to talk about the Maia 400.
Saurabh: That will be soon too.
Okay, excellent. All right, thank you.
Saurabh: Thank you.
