The future of inference is asynchronous

Today, I'm talking with Neil Movva, co-founder and CEO of Sail Research. Sail provides inference and infrastructure for long-horizon agents. Neil, it's really great to have you. Thanks for coming on.

Thanks for having me.

I'd love to kick off the discussion by talking about how you see the shape of the inference market. I think by now it's clear inference is this huge and rapidly growing market, but what are the workloads we actually see? What is the distribution of work that is, for instance, focused on throughput versus needs very low latency?

Absolutely. I think we can agree that one of the use cases that really kicked off the inference market was coding. It has extreme PMF today. It's clear that coding has changed forever at this point. And coding is what really drove a lot of the inference demand and structure of the providers you have today over the last year. Specifically, the coding market demands low latency. It's a very interactive use case. There's a user, a programmer, who is in conversation with an agent.

And they're typically talking to that agent every minute to five minutes. That's the window of turns at this point. And that is the core loop that all the current CLI harnesses, like Claude Code and OpenCode and otherwise, are built around. I think that's what has pushed the inference market to focus on speed. If you want to have an interactive experience with a high-value use case like coding, you're going to focus on speed.

You have Claude Code having fast mode. You have inference providers like Fireworks and Baseten and Together really focusing on speed—tokens per second, that is. And there's always going to be more appetite for that kind of demand. But in the last couple months to about a year now, I would say, we've seen a new type of use case, which is these agents that can run autonomously or in workflows for maybe much longer between human interactions.

Instead of running in a loop with a human providing input every five minutes, the agent will run a workload for maybe hours, maybe days, and eventually even weeks. We see this with, for example, OpenAI chasing math problems with very long-horizon agents. We see this with OpenAI's top internal users spending thousands of dollars per day on tokens. I find it hard to believe that those thousands of tokens are being spent where the human is in the loop on all those tokens.

These are some form of background agent that's being used here. And that's what really excites me, is this rise of background agents. There is a pretty tight bound on how many tokens one person can consume when they're at the laptop talking to the agent. There is no bound on background agents that are running against hard problems for hours, days, and weeks.

What would be helpful is maybe zooming back just a little bit and talking about the trade-off between throughput and latency as well.

Absolutely. We talked a little bit about how current inference is pretty focused on these real-time interactive use cases. And what that really means is you are focusing on having the language model produce output tokens as quickly as possible. That is the core workload we're really dancing around here. But maybe we should zoom out again and ask about what a language model is. What is the computation involved there? There's a transformer, which is, of course, a large neural network, which has billions, if not trillions, of parameters at this point.

And you can take the inference workload and break it into two phases. There's prefill, which is processing a large prompt that the user has put in, or it could be the output of a tool call. It's a lot of text that's coming in, thousands of tokens at once. That can be processed in parallel, and we call that prefill. The term comes from basically you're filling the KV cache. And once you've filled those tokens, you never have to compute on them again. You're just recalling them from memory.

Then there's an output token phase where you're producing output tokens one step at a time. And we call that decode. We are decoding the answer from what we've prefilled in the past. And I would go so far as to say decode is the only fundamentally expensive thing in transformers today. It pushes the GPU into a regime that it's not necessarily originally designed for. Because what happens during decode is you have to read all the parameters of the model, billions or trillions of parameters, at least once for every token you're producing.

And so we end up being bottlenecked by the time it takes to load the weights, these trillions of parameters, into the GPU, perform the computation, and then cycle that iteratively. And the GPU is basically spending all of its time loading the weights. It's not spending its time doing the compute itself. The compute is cheap compared to the weight loading.

That's why we get this idea of these two phases. The GPU can be either compute-bound, where it's doing a lot of work during prefill. All the tokens are sent in at once, and it can amortize the loading of the weights across maybe thousands of tokens that it's computing. So it becomes compute-bound. And the other phase, where we're processing one token at a time per stream, becomes the memory-bound decode phase.

Why can't the GPU make the memory faster? Well, this is an interesting question. This is one of the longest-standing challenges in computer architecture: we have always found ways to advance the speed of compute faster than we can advance the speed of memory. As Moore's law basically gave us more transistors per square millimeter, we were able to pack on more logic and do more work per cycle of compute. But memory is an entirely different industry. In fact, you have different fabs, different companies specializing in memory. And the fundamental structure of DRAM, of memory that we use for computers, is so different that it becomes hard to just incrementally speed that up at this point.

And so what you end up with, then, is essentially a whole industry that is bottlenecked on memory bandwidth.

Yeah. That's a large discussion we can have. We could talk for hours about memory. But yes, we end up having this ratio. There's a ratio between the amount of work—compute, or arithmetic, really—that we can do per cycle on the logic die compared to the memory we can fetch from memory chips that are stacked alongside the logic die.

I call it the roofline ratio, which is basically to say, what is the minimum amount of work you must do per byte of data you fetch in order to be compute-bound? If it's more expensive to fetch data than it is to compute it, how many operations must I do per byte of data that I fetch to be compute-bound?

And that number for NVIDIA GPUs has been remarkably stable over the last few years and across their families of chips. It's usually in the mid-hundreds, so 400 to 600 or 800 or so, a pretty tight band. And there have been other accelerators in the past that have gone well above 1,000. That's a really compute-focused accelerator. And there are CPUs, which don't really focus on compute in the FLOPs sense, but more on branching and control flow, that have a much lower roofline ratio, which is to say between 10 and 100, typically.

I'm curious if you've seen a shift on the hardware layer for this workload as well. I think that the changes we've seen workload-wise ripple through. So you have initially maybe changes in the way that you distribute the workload coming in, changes in the way you write kernels, and then eventually it flows down to the hardware level too, where today there's a lot of ASICs: some focus on high throughput, some focus on low latency.

I think the ASIC market is very interesting to look at as well. All hardware architects want one thing to make a great chip: they want specialization. They want control, or tight definition of the workload, and that allows them to build more structure into the chip that suits that workload, broadly speaking. And in deep learning, that, you could argue, has already been done somewhat by NVIDIA, in the sense that they've added specialized hardware to the chip that is uniquely suitable for deep learning or modern deep language models.

And that hardware is the Tensor Core. The Tensor Core is a very important part of the foundation of what makes a GPU great at matrix multiplication. The GPU initially was thought of as a parallel machine, and so any parallel problem should fit on this parallel machine. And that is true, but the GPU historically was a vector machine, vector being that there's one dimension of parallelism here, and then you can just stamp out more vector units.

That is a reasonably good way to compute matrix products, but you can do even better. You can build a two-dimensional structure on the GPU that, for large matrix multiplies, turns out to be even more effective at computing tiles of a matrix multiply with much higher arithmetic intensity and much more ratio of compute to memory movement. And so NVIDIA started putting those into the chip about 10 years ago, and those have really grown on the chip.

We started with small systolic arrays, which are the structures we use for matrix computation, and we grew them over generation over generation to where they are now, which is maybe fairly large-size systolic arrays. And there are some other architects of chips who saw this pattern and said, "What if we do it all the way? What if we build a massive systolic array onto the chip?"

Systolic arrays have an interesting property: the data movement, the number of wires and energy you need to move data onto the systolic array, is proportional to the perimeter of the array. If you think of it like a square systolic array, it's going to have data movement proportional to N for an N-by-N systolic array. But then the compute on that array is N squared. And so you have this natural ratio that the arithmetic intensity of the systolic array is increasing as you get to larger N.

There's this goal to make that as big as you possibly can. That is one of the ways to get really high efficiency of compute on a chip. And so to zoom back out to ASICs, there are people who have been trying to build very large systolic arrays. You could argue that the TPU was the first to go big on this. And nowadays we see newer chips like what Etched is doing, what MatX is doing, that go for larger systolic arrays, and they get more efficiency that way.

Just to make sure I'm understanding you correctly, then: if you had, for instance, a gigantic block of compute, you might actually be less efficient on the bandwidth side because you're not allowing enough transference in between. And so if you instead have a split systolic array, then you might have much better communication on your memory bandwidth.

Yeah, I should define these a little bit more. I think that you want the arithmetic intensity, the ratio of compute per cycle to bandwidth moved per cycle, to be high. Fundamentally, memory is expensive, memory bandwidth is expensive, and compute, I wouldn't say it's cheap, but it is relatively cheaper. And so higher arithmetic intensity is a design where you're going to get higher efficiency. You're going to get better performance per watt, performance per area. Everything is going to be better at higher arithmetic intensity.

But you don't control the workload always. The chip architect does not design the shape of the matrix multiply that the model wants. And so the challenge for architects has always been to balance flexibility of the chip with the efficiency that is inherent to larger fixed blocks. And this is most concretely felt in ASIC design for AI with respect to the size of the systolic array.

And to your point about splitting the systolic array, that's absolutely a thing people do. Essentially what I'm saying so far is that big systolic arrays are good, so why not just make it very big? Well, if you have a small matrix multiply that sits on a large systolic, you're going to be wasting a lot of the area and compute is going to be idle. You're going to be padding it with zeros and wasting precious area.

I think a lot of this is dependent on what workloads end up looking like, right? And I would love to hear how you've seen workloads change even in the last year, but I'd also like to hear your thoughts on what it's going to look like in the future and how much of that market you're interested in. Because, as mentioned with robotics, I think that the market is not stagnant right now. What we're seeing right now is a checkpoint of, okay, we're moving from person-to-agent work into agent-to-agent work.

Yeah. So there’s one theme that I’ve followed for many years now in all the machine learning, which is this idea that most patterns that you were trying to model in a deep language model, or even traditional machine learning model, most of these patterns are very sparse. You want to have a lot of world knowledge, but on any given prediction, you are not going to leverage all the world knowledge that you've trained or compressed into a model, as a high-level idea. So there's some inherent sparsity to these problems.

How do we capture that sparsity? Well, it's been a challenge. We started by building these super large models, heavily overparameterized. For example, people were training hundreds of billions of parameters of dense models, meaning on every forward pass through the model, you're activating hundreds of billions of parameters. All the parameters in the model are being used to make a prediction. But we know intuitively, and also if you mechanically interpret the activations or the projection of these weights onto this input, you'll find that most of the parameters are not involved in a single prediction.

And we've started to formalize that about two years ago with what we call mixture-of-experts models. The idea being that you can take a very large model and have it, for itself, figure out which subset of its parameters are most relevant to every token, for every prediction that is going through the network. And that is a really great way to capture structured sparsity within the model.

We've gotten pretty good. I'm not familiar with the tricks. I'm not a pretraining guy. But there are many tricks that pretraining people have figured out to have the model dynamically select which subset of its parameters are going to be used in every forward pass, or on processing each token. And so that's been a major change for hardware. It means that the systolic arrays are not just going to get bigger and bigger infinitely.

Now we do have to consider that maybe we can get more granular for each expert that we're mixing over. And so maybe the array shape is not going to be just monotonically increasing, but rather we're going to get to a point where we have, yes, trillions of parameters, but only a small subset of them active on every forward pass. We are there today, and that's what's been a challenge for people who build really large systolic arrays. It's hard to fill them with small experts.

I think mixture of experts is one big change. And then maybe even higher level than that is the change to multi-agent-type workloads.

Right. There's two levels to talk about here. There is the mechanics of the language model itself. Model architecture is very near and dear to my heart. We want to be maximally efficient when we build software and hardware for these language model structures. But at the same time, there's this larger pattern change around what we talked about earlier, which is moving from interactive chatbot-type use cases to more asynchronous, long-horizon agent use cases.

And that transition has only really been quite recent. Models have to be good enough to trust to run for hours on their own. And we're just now seeing this massive renaissance in personal agents. It feels like we see one of these every year, but now they really work. And so they're on the cusp of, or actively achieving, product-market fit. It's really amazing to see. And I think that that market is really interesting because it'll be most people's first experience with really proactive agents. That's the trend I'm most excited about right now.

Is that primarily the type of work that you'd be targeting? Because just to say that a workload is multi-agent doesn't necessarily mean it's latency-insensitive, right? In fact, much of multi-agent work is actually very latency-sensitive. Jalapeño is actually built for latency for this reason. And so what do you look for when differentiating workloads that are not latency-sensitive versus ones that are?

Really, the axis I'm looking at is more proactivity. What can the agent do without you asking for it? There's this glib statement we like to make, which is that the best latency of all is none. If you can proactively predict what the user wants and go after essentially doing sleep-time compute to get that work done ahead of time based off what you know about the person, your priors on that person and their use case, well, that's the best way to surprise and delight a user.

And our goal is to make compute so abundant that that kind of proactive computation becomes essentially cheap enough that you can do it much more often. And the reality is that most of these proactive, I guess, threads are not going to resolve or are not going to result in anything visible to the user. Most proactive threads will not result in a Navier–Stokes proof. They will not do anything useful. But we want to encourage that exploration.

There is a lot more you can sample from a language model than just hoping that it gets the right answer in response to a user's demand. And so there are some early ideas here. I wouldn't say that they're super well fleshed out yet, but the idea of sleep-time compute, the idea of going after anything that requires lots of computation and search in particular, we think are really productive things to mine. Not there yet, though.

Yeah. What would you say are Sail's ideal workflows?

So we love deep research—hinted at it just now. We think that deep research is one of the problems that scales best with compute. Search is a verifiable problem in a lot of cases. It's easier to verify that you have a solution that matches the predicate that the user gave you than it is to sample that on your first try.

And so we've done a bunch of work on benchmarks like BrowseComp to show that you can, through compute scaling, specifically parallel compute scaling, launch many subagents to cover a massive haystack and find really, really specific needles in that massive haystack, all through parallel intelligence. And so we're really excited about test-time compute scaling over hundreds of agents working on the same kind of high-level problem being broken down into subagents and converging again.

I also wanted to shift gears a little bit and talk about compute just generally, compute capacity. So I think one big question that everybody has is: how are you getting more compute onto the platform, just to even start off with? And what does that distribution look like today between—not necessarily who your providers are, but what the shape of your providers are—and how you end up working with those relationships when the market is just so, so supply-constricted?

Yeah, well, we very much think that inference as a category is about supply. There is not really a trick there. You have to just be more diligent on supply and find some alternative ways to get supply. That's what's going to make winners in the inference space. I will start by talking about the obvious market, which is the market for NVIDIA chips.

NVIDIA is still the best chip manufacturer in the space. They are absolutely the most scaled. They have the best software out of the box to serve language models and really all AI workloads at this point. And to challenge them, I guess you have to be very careful on what regime you want to challenge NVIDIA on. They're extremely good at what they do.

That said, the problem with NVIDIA is that even as good as they are and as scaled as they are, they're still very finite in supply. We cannot really double the number of NVIDIA chips coming online in the next year or two, or maybe even three. It is hard to imagine that we can outbid the most well-capitalized players in the market for the finite NVIDIA supply. And I'm referring to here, OpenAI and Anthropic. I think the hyperscalers and the big labs are going to continue to buy as much NVIDIA supply as they possibly can.

Now, NVIDIA is strategic about this. I think they're careful not to let one or a few small monopolistic kind of providers soak up all the supply. They understand that their chip supply is actually kind of the lifeblood for startups and anyone who wants to enter the AI market, really. And so I think they're doing their best to balance supply.

But at the same time, the reality is that Anthropic makes a lot more money per GPU hour than any open-source provider does. They seem to be willing to pay for a Blackwell hour, a B300 hour. They're willing to pay something like $15 to $20 per hour. And if you compare that to what open-source providers can make or provide, it seems like it's hard to make a market above $10. And so given all that, we have to find some way to explore alternatives that are underexplored.

And for us, that has meant alternative NVIDIA chips. We are really excited about one of their chips called the RTX Pro Blackwell, which is a consumer chip that was kind of repurposed into server use cases. We're very good at using that chip. I think we have written some of the best kernels for what NVIDIA refers to internally as SM120. Slightly different chip, different memory technology. We sidestep the HBM shortage there. That uses GDDR instead of HBM. Simpler packaging, more abundant in the market.

It's great. It's a really good chip, but it has some quirks that are not well discussed or explored in the open-source inference engines or even by NVIDIA themselves. So lots of room for us to do better on that kind of chip. And of course there's alternative vendors who are doing a really good job of playing in the regimes that are not totally dominated by NVIDIA.

And so again, there's these air-cooled PCIe cards that can be really well done by AMD, Intel, maybe some others who are coming to market. Qualcomm has some ideas here. And they just sidestepped the bottlenecks that NVIDIA would run into as well. And so most importantly, that's memory today. You cannot be using HBM and have a good time.

Yeah. How do you think about tackling this problem from the business standpoint? So one component is picking chips. But I don't know if you guys are really in the data centers choosing a chip.

So today, we are building our first data centers, but that is a long and arduous process. We have a lot of respect for the problem, so we are taking our time. We will have this done. We have some test machines already, and they will be online by the end of the year, and we are serving on some of these test systems already. But to really scale this play will be well into next year.

On the other hand, though, I think we think the data center design is going to evolve quite a bit as well. Currently, the optimal thing to do if you have access to NVIDIA chips is to put them into a large monolithic data center. Maybe you want to put 10,000 chips in a data center. That's something like 20 megawatts, and you should try really hard to go find 20 contiguous megawatts in one facility that's high reliability, and you should rack those chips and collect profit. It is a very good thing to do if you have those chips and you have access to that kind of powered shell capacity.

If you don't have that, well, maybe you should try to do something a little bit more scrappy. We like the idea of small modular data centers. We think that the future for inference is that we don't need to have large contiguous supply. We are very happy with small data centers that are sub-megawatt in scale, maybe just a few hundred chips, maybe not even a hundred chips. And so if that's the future of inference, maybe we should work with many small pools of compute instead of one large monolithic compute pool.

Of course, any distributed systems engineer who just heard what I said would be very stressed about the overhead of managing so many small pools of compute. They're all going to fail at different times. They're very hard to even manage, from a Kubernetes or similar perspective, all these small, small pools of compute. But we think that's the kind of trade-off that we should make. We should be more aggressive in that direction because it is the area that is least likely to be explored or least likely to be easy for other players to do well. That is where we should build our IP.

It does feel like there's more pressure to build these modular data centers even today. I mean, I think it was SemiAnalysis that said by the end of 2028, they estimate that about 30% of data center capacity in the US is going to be for modular. I think NVIDIA right now is actively piloting some modular sites between five to 20 megawatts. It depends on what your definition of mini is. I don't think they're thinking about one megawatt just yet, but it does seem like an intuitive direction to push on. Why are people not pushing even harder?

If I'm trying to build a data center today, it's first very hard to secure capital for that, and a smaller pool of capital is easier to secure. It's even harder, arguably, to find land and to find power. And so the more you scale it down, if somebody is able to solve the distributed systems question, which it seems like you're tackling—and eventually I think anyone who's working compute will think about tackling—why is this not an even bigger market?

Great question. The other challenge with these modular sites is that it's going to be very hard to get them to Tier III or Tier IV data center uptime, meaning Tier III and Tier IV are the highest tiers of data center reliability. They have redundancy across power and cooling and internet. They're designed for maybe three or four nines of uptime in most cases. It's going to be hard to deal with these modular sites. They will not have redundant power most often. They will not have redundant cooling, or maybe they'll have minimal redundancy there. And so we're kind of commingling two issues here.

One is that, yes, modular sites are going to have a smaller functional unit. It's going to be hard to put 20 megawatts inside a single container or modular site. It is going to be closer to a megawatt per pod. And the other issue here is that, yeah, if you put NVIDIA machines inside these modular data centers, well, the NVIDIA machines are too valuable to tolerate that kind of downtime. It is going to be very painful to sell this cluster of premium chips with not premium uptime guarantees. And that's one of the challenges that people are trying to solve right now.

Can I ask why that is? If it's simply a culture thing of people who don't have enough capital to build out really large data centers, and so they end up having less redundancy for power and cooling and such. Because I would imagine that scaling down the size of the data center—I'm not sure why that would correlate with necessarily scaling down the redundancy or the sheer quality of it.

I see. Well, why are you scaling down the data center size in the first place? You're doing it because it is cheaper to build, for sure, but also because it lets you be a little bit more scrappy. I keep coming back to this term. I'm really envisioning here nontraditional sites. This is going to be like a warehouse in the middle of nowhere, or an abandoned steel mill or an abandoned brewery. Something like that is going to be a good site for these modular data centers.

They maybe have grid power, but the grid power has some curtailment provision, meaning it can be turned off for so many days of the year. These are not traditional data center sites, and you're going to have to deal with that somehow. So modular is following the long tail of powered shell capacity. It's not so easy to make that powered shell capacity suddenly super high uptime.

The other thought I have on that is training demands contiguous capacity. People want to have 4,000 or more chips for training. And you're going to have a hard time giving them that with these modular sites. So I do think modular capacity is primarily for inference. And it is primarily for people who are not afraid to accept the lower reliability and work around it in software. And as a young startup that's mostly focused on software, that's exactly where we choose to invest.

Yeah, be brave.

Yes, we have to be. There's no other option.

Going by stack, what does it look like to try to tackle heterogeneous compute, potentially power failures and less redundancy, managing the Kubernetes on this? Just take me through that step by step.

We're spiritually inspired by Google. Early Google was all about this. Instead of going to buy mainframes from IBM or similar, really reliable data center companies, they would build their own from commodity hardware and put them in warehouses often, and accept that the machines are going to have a higher failure rate. They're not going to buy hardware redundancy to get to five nines of uptime. They will buy one-nine-of-uptime machines and route around the failures in software.

We think this is a very powerful idea. It is a very nice way to understand that the extra nines you're buying in hardware are very expensive. And if you can do it right in software, hopefully you can transmute the problem to becoming essentially a linear instead of a reciprocal problem. Meaning I should be linearly happy for 90% uptime versus 99% uptime versus 99.9%. And therefore, if the 90%-uptime data center is half the cost, I am 100% going to buy the one nine of uptime.

That's the high-level goal. Again, this is all software that has to be done really well. And also it's enabled by our workload. If you care about P99 latency, well, it's in the name. You can't have a tail failure one in 10 times. You can have it once in 100 times, but even then your customer's going to complain. It's very hard to make P99 latency tightly controlled if you're going to go do these exotic and less reliable individual sites. We have to be able to recover from a failure more efficiently, and recovery will always involve extra time. We will be durable, but we won't be P99-controlled in a lot of cases.

I'm doing this only because I believe this is the correct trade-off for agents. Agents don't get work done in a single turn. They get work done over a trajectory. And the trajectory will happen over minutes or hours anyway. So our view is that if an agent that's running on a machine that's on an unreliable data center suffers a failure, we will lose that turn's progress, but we will have all the state we need to recover and move it to a different site and immediately resume the computation. Now, immediate here means within 30 seconds. So we will see a blip of 30 seconds of delay on that one turn. But if it happens infrequently enough, it's unlikely to move the overall trajectory time a substantial amount.

If I come to you with a workload, and normally I would like it done in a minute, but I say, "Hey, I can stretch it out to an hour," how do you position that workload in the queue? How does it go through that pipeline that you just walked through?

For something like that, where we have a lot of flexibility, we have a tier of our service we call the Flex tier, which is designed exactly for hour-type flexibility, or 30 minutes plus. And I think in that tier, the most important things to do are a couple things. First is batch size. We really want to make sure that any requests we push through our Flex tier are running at the largest possible batch size that we can achieve.

And the reason for that is what I mentioned way earlier: you want to amortize the expensive thing in a forward pass, which is loading the weights. You want to amortize that over as many requests as possible. So batching is really important. The second thing is KV cache. Of course, we don't want to redo compute that we've already done. And so it's really important to us to hit the KV cache as aggressively as possible, but the KV cache for a given request may be tied to a specific machine.

And so we have to wait for that machine to become available. And having this flexibility on workload scheduling allows us to hit that KV cache much more reliably. We tend to have some of the best cache hit rates across the board on all of our service tiers. And then maybe the last piece here is the Flex traffic that is focused on longer term or less sensitivity to latency, but a strong focus on scale and cost.

We should run those requests on other chips that will not be competitive with NVIDIA on tokens per second or interactivity, but will be very competitive on cost per FLOP. Essentially, NVIDIA is the king of two things: low-latency kernels, period. You can program the chip to move data around with low latency very flexibly. And number two is they have really, really great interconnect between chips. NVLink is a triumph.

It's really impressive what they've done, and it's actually a pretty core idea to Jalapeño and Etched. They're both very correctly investing in interconnect between chips for low latency. So that aside, yes, you want to move your more flexible traffic onto alternative chips that are competitive with NVIDIA on raw FLOPs, but maybe less so on the interconnect and generally low-latency kernels.

And what do the cost savings for this end up looking like? If I come to you with a workload that normally costs $100 through a traditional neocloud and I say, "Hey, I'm willing to tolerate definitely a lot of delay on this one," how much could you save me, and where does that cost savings come from? Is that purely from the hardware shift?

I think I usually quote around 5–10× savings, and I factor that into two improvements that stack, each between 2× and 3×. The first one is just packing larger batch sizes, as in we have written the entire inference engine to be much more focused on MFU, FLOP efficiency, model FLOPs utilization.

And so we will achieve twice as many or three times as many tokens per GPU minute on our serving config. That's the first axis. The second axis is using alternative hardware that would not have been suitable for low latency in the first place. And that is another maybe 2–3× improvement on a cost-per-FLOP basis.

That's very impressive.

It's great. It's nice to be able to use the machines in the place that they have comparative advantage. We've actually published some of our work in how we analytically look at the fleet of hardware available to us. We are fairly agnostic to the chip provider. We think that we can write great kernels for almost every chip. And we are experienced with at least five.

We work with NVIDIA. We work with AMD. We work with Intel. We work with TPUs and Trainium. We love all these chips. There are no bad chips, only bad prices. And so it's our job to price those chips and figure out, for this workload, here's the optimal chip.

How do you manage this control plane?

The first trick that we have is to have respect for the problem. I keep coming back to this. Many people think that inference is about kernels, or about knowing all the CUDA lore, and knowing what a tcgen05 is versus a Hopper WGMMA. It is a little bit about that, but really, you should start by respecting the control plane. Many other companies don't do this correctly.

And so the first thing I did when starting this company was hire the best distributed systems engineers I know. For example, my co-founder, he does not come from AI. He had not written kernels seriously before starting this company with me. He comes from cryptography even, but he is the best programmer I know. And great programmers have always loved systems problems. Distributed systems is the historically hairiest thing you can do as a programmer.

And so I built the team around him and all of our best friends from school who have worked on these really hard systems that need to be reliable, that must be rock solid. People who worked at trading firms or other sorts of "must stay up" use cases. We found really, really great results working with them. And so we have about 10 people on that team right now. It is still the largest singular team in our organization: the control plane, distributed systems, core inference service.

Systems is very difficult, but at its core, I think it is still very much a software problem. And so do you see this becoming much smoother, both for Sail and potentially for other neoclouds or anyone else who wants to pursue this path in the future because of that?

Because it's a software problem and agents will help? We would all love that to be true. I think that this is one of the hardest problems for agents to help with because agents really benefit from verifiable problems. It is hard to verify a distributed system because the most challenging bugs only happen at scale in production. And so it's very hard to even reproduce some of the issues that we see.

If you have spikes of traffic, the exact pattern of arrival dictates how your scheduler is going to handle them. In fact, the content of the requests even impacts the workload. For example, KV cache is stateful. All of our workers are stateful all of a sudden. Well, that's very painful to deal with because now a request could take easily three times as long if we don't hit KV cache versus if we hit it. And that is something that is just hard to replay in a simulated environment.

And remember that the GPUs themselves are very expensive. You can't just replay every night's traffic the next day on a fleet of GPUs. I mean, you can, but it's not the thing you would do a thousand times over with an agent. And so I do think distributed systems, by definition, are a little bit hard to make verifiable because the interesting stuff happens only in production.

Contrast that with kernels and engine work, which is absolutely a verifiable problem. It's self-contained. You can test it, you can iterate rapidly. And so in theory, that should be very much solved by agents. We still find that hiring expert kernel programmers and training them to really work well with the agent—the centaur model—is way better than just asking Claude, "Please improve the kernel. Make no mistakes." So we have found hybrid, essentially human-agent collaboration to be the best.

To me, this feels like an engineering question, not a research question. It is clear that with enough data, the agents will become very, very good at solving distributed systems problems. So it might be a very long time horizon, but it does feel it will get solved eventually, right?

First of all, one of the statements that I agree most with Elon Musk on is this idea that there is no research-engineering split anymore. It is all engineering. Everyone has a bottom line and they want to improve it, including researchers. They have a number or an eval they want to make go up. Broadly speaking, we're all in the same position. There is no monopoly on creativity for researchers.

I want every engineer to be as creative as a researcher considers themselves to be, and I want every researcher to implement the work that they choose to pursue. There is no fence between the two that you can throw things over. First statement on research-engineering: everyone at Sail is an engineer.

The second thing is, is it possible for agents to automate? Fundamentally, it seems like it. I don't think, though, that distributed systems are easy to verify, and that will make some challenges. Let me say it this way. It is really painful to debug distributed systems code that you did not write yourself. It is very, very painful to not know where the bug could possibly exist. Of course, the agent will help you, but if you don't have a conceptual understanding of how the system is put together and what trade-offs were made getting there, it's going to be very difficult to fix issues.

I think the right way to think about programming this kind of system in production today is we are going to do our programming on the whiteboard. We will conceptually understand what's going on, and we will absolutely have rigorous understanding and, as much as possible, simulation of what we think the machine is doing. But if we don't understand what the machine is doing, we're going to have a very hard time debugging it or designing it to grow in some way. But the execution of that plan onto code, agents are very reliable on. You can almost think of them as whiteboard-to-code compilers. I like that framing.

So this leads me to a kind of common-ish or often-said phrase, which is, "Compute is a commodity," or, "Compute will become a commodity." I think at current state, it's very clear that it's not. But I think the future that people envision is that you could abstract away perhaps the actual hardware layer, which I think it's clear is getting more heterogeneous, not less, and maybe consider just GPU hours or accelerator hours in the future perhaps to be the currency or the commodity. Do you see a future that happens? Because I think in order for it to—what Sail is doing, it would be very, very critical to that, like managing all these distributed systems.

Yes, that's a great question, and again, one of the things that's really been a struggle for the industry. We all want compute to trade more liquidly like a commodity. Oil can trade like a commodity. Gold can trade like a commodity. But for some reason, GPUs do not. And the reason is, well, I think the biggest one is that we are not capturing the increase in value that a GPU has when it's in a larger contiguous quantity.

A thousand GPUs in a single data center is worth more than a hundred GPUs in the same data center. And 10,000 is even more than that. You see this with, for example, Colossus. The SpaceX cluster regularly rents for twice or three times the price that you would find on the market because they are the largest contiguous cluster in the world, and they absolutely can command that premium. If you are training a model, you will pay the premium for contiguous clusters. That is purely on just quantity alone.

Then there's a second part, which is reliability. There is no standardization of reliability or uptime. The closest thing would be SemiAnalysis ClusterMAX ratings. That's all people have to go on, and it's a signal that there's demand for standardization. And again, you see this showing up as well on the two platinum providers on ClusterMAX, Nebius and CoreWeave, charging the highest premium on the compute because when you work with them, it does tend to just work.

The compute should be more fungible. There is a long road to get there. Our view and our role in this market is to provide some level of liquidity on the buy side, meaning we want to be the most liquid buyer of compute in the world. We will take any chip anywhere in the world for any duration of time.

And so every chip can be priced at the Sail price, essentially. Now, it will not be the highest bid because we are going to take advantage of the fact that we are more liquid than others to get some spread in our favor. But at the same time, there's no stranded compute for us anymore. We will consume every capacity, every quantity. And that's why it's worth investing in the distributed systems to essentially handle hundreds of small pools of compute.

Are you interested in eventually expanding to larger pools of compute?

We will, of course, buy larger pools of compute. All I'm saying is that it is more expensive because they can do more things. But we are so good at using small pools that we should take that advantage and turn that into more spread for us.

How do you think about the shift between the training and inference paradigm? So one shift, just very, very high level, is that a lot of the workload has now shifted to inference. But even within training, if you separate pretraining and post-training, now a lot of the focus is also shifted to post-training, and a lot of post-training is just inference.

Yep, that's right. Yeah, post-training, I mean, RL loops are almost 80% inference by compute or by cost, however you want to look at it. And so we absolutely see ourselves playing a role in that as well. I think what makes inference for post-training special is that the weights will change frequently, and so it's really important that we support things like LoRA adapters and fast-changing LoRA adapters.

And so we actually really quite like what Tinker has done in this space. I think Tinker sets the bar for quality and reliability of the backward passes. The training piece is pretty tricky to get right. I think they do an extraordinary job of doing that. I have no interest in doing backward passes until I can do it at that level. And I will let Tinker continue to do the backward passes at the level that they're really great at ensuring quality on.

Instead, I will focus on being a downstream provider from Tinker, meaning once you've trained a model, or even while you're training a model, you want to sample from a new checkpoint many, many times. We can do the sampling really efficiently. And we can, in fact, integrate with Tinker in the sense that we can take checkpoints that are coming off the Tinker trainer and load them into our serving stack rapidly, within minutes.

And so you can build an async RL loop entirely around Tinker plus Sail, where Tinker is doing the backward passes. And every time you get a new checkpoint for a LoRA adapter, maybe every five minutes or 10 minutes or so, it will be sent to the Sail workers, and the Sail workers pull it, and you're sampling from the new checkpoint within a minute.

And you manage the integration of that?

We do. Tinker makes it relatively easy. They're great. And we are happy to do forward passes underneath Tinker's backward passes.

That's very impressive. Do you see more of this shift continuing? Post-training is very big right now. It's not a guarantee, right? We might find new types of post-training regimens that aren't just RL, but at least for RL, it seems that we're focusing more and more on inference. And then on another level as well, it just seems generally with test-time compute that if you throw more inference at the problem, your results will be better.

Yeah. We think inference is just about surpassing training. It's very hard to get good numbers on this when a lot of the compute is being consumed behind closed doors like OpenAI and Anthropic. But it does feel that inference has surpassed training, but the market is not caught up to that. And what I mean by that is data center operators are very conservative people. They do not want to make strong predictions on what is the split between training and inference in the future. And so they build for the superset of both of them. They build these large contiguous clusters that are suitable for training. And of course, you can carve something out for inference.

I think that we already see that maybe not being the right strategy. If in RL, you often separate the training cluster from the inference cluster. You can do this across two separate machine, I guess, pools entirely because the data bandwidth between the two is not significant. You're sending small weight deltas. For example, even if you're doing a full fine-tune, frequently most of the parameters are not changing between checkpoints, between steps. And so if you can delta-compress checkpoint one versus checkpoint two, you are able to send this delta over a pretty low-bandwidth connection.

And then of course, the actual sampling, what you're sending to the inference service is essentially a large RL environment, or we have an RL environment running somewhere else, and what we need from the inference service is a bunch of rollouts, which are just tokens or text. And so we're feeding back fairly low bandwidth between the training cluster and the inference rollout cluster.

So I see these dichotomizing. There is some interest in maybe having the trainer match the inference worker as closely as possible from a numeric standpoint. There are some people who believe that you must have zero KL divergence between the trainer and the rollouts. But it's above my pay grade to say whether that's really important. All I can say is that async RL seems to work, and that's kind of a biased delta in the sense that you are actively rolling out stale weights while there's a newer policy available.

And so the policy difference between the numerics being different for the rollout worker and the trainer don't seem—at least that's unbiased. It's just quantization that's causing the difference. And so having unbiased policy difference versus biased policy difference that seems to work, I'm of the opinion, and it seems like there's some empirical evidence that shows that it's not a big deal to chase zero KL. But I'm not super educated on that, so I won't say more.

And then as far as the specialization goes, so some portions of it are just different companies doubling down on different parts of the workload. We also see this both on the software, the kernels layer, and recently, I think, on the hardware layer as well, where there's interest in, for instance, PD disagg. Yeah. Could you talk a little about that?

Yeah. So PD disagg is a really interesting use case. I mean, it makes the system so much easier to reason about. You can think about the prefill specialization very differently from decode specialization. And the core idea in PD disagg is we don't want to interrupt decode batches with prefills. Prefills can arrive at any time, and they basically stop the generation of decode tokens at any time. And so if you're trying to maintain a consistent output TPS on the decode batch, well, disaggregating the prefill is a good way to do that.

But you are also kind of leaving some capacity idle. Your prefill worker is waiting for essentially work to come in, or it has a queue and you're basically queuing the prefill, and it's not great either. And the biggest challenge of PD disagg is, of course, the dynamic split between prefill and decode. You want to be able to control that ratio quickly.

But most people today are doing PD disagg across physical machines. And so reconfiguring a P worker to become a D worker is not fast. It frequently takes minutes even if you optimize it. And so there's some interest in maybe doing more fluid forms of PD disagg. Perhaps you could do PD disagg within a node or maybe even within a GPU. There's some work on doing PD disagg across SMs on a GPU, which we find very interesting. We've done a little bit of that work ourselves. And these are all in the service of being more fluid about the allocation of prefill to decode work.

And this is still all on homogeneous hardware. PD disagg today is almost always done with the same chip for prefill and decode. And this is something that we think will change in the next year or two, is using a memory-bandwidth-specialized machine for decode and a compute-specialized machine for prefill. The arithmetic intensity roofline you need for the prefill worker could easily be 2,000 or more, which calls for non-HBM technology, and then the decode worker could be HBM-focused.

Is this something that Sail thinks about when considering how to balance across different workloads? Roughly, is this a distributed systems problem?

Yes, although this is most relevant for maintaining high output TPS. I keep coming back to the idea that output TPS is the most expensive thing in inference. And so if you want to serve at 100 or 200 tokens per second of Kimi or similar, you must have really good speculators, which means running a large lookahead window. And you must also have PD disagg. There's no other way around it.

If you don't want to do that, if you want to serve at 20 to 50 tokens per second, for example, which is where OpenAI and Anthropic frequently serve at, and where we choose to serve at for our fastest tiers, we think that PD disagg is maybe less important. You can get a lot of the same benefits from chunked prefill, which lets you pace the prefill over the course of a decode batch or over many decode steps. And so there are alternatives that can be more efficient.

If you need to do PD disagg, you need to, for example, invest in backend network between the prefill machines and the decode machines, InfiniBand or RoCE. And those are getting kind of expensive now. And so all we have to do is respond to these market incentives. We think HBM is expensive; we're going to de-emphasize HBM. We think InfiniBand is expensive; we're going to de-emphasize InfiniBand and see if we can make things work over cheaper, lower-bandwidth interconnect between GPUs.

So I wouldn't rule anything out, but generally speaking, yes, PD disagg is really important for low-latency tokens today. It's not as important for high-throughput tokens. High throughput is all about filling every cycle on the machine with work, and there's ways to do that that don't involve PD disagg.

Do you get concerned about economies of scale, particularly as there's a lot of centralization of compute today from the labs, but also from the hyperscalers? And if the idea that smaller players will have is, "Well, we can just focus on lower ends of the market," the problem is you need your margins to be higher to offset not having economies of scale, right?

Well, very funny you mention that. I actually believe that there are extreme diseconomies of scale to inference or to compute. It's very hard to build a large data center. It is not easy to do even a 10-megawatt site, let alone a 100-megawatt site. Everything from political opposition to the fact that you have to do net-new construction, getting that done in the short term is very, very difficult and painful.

It is far easier to retrofit an existing small site, especially if you're working with one of these modular data center providers who has shipping containers ready to go. Provide power, provide hoses for cooling, and you're good to go. There are lots of creative things you can do there, but they won't be large contiguous sites. And so we are now opening up this category of supply that never existed in the market before.

No one ever thought about building these tiny, tiny data centers in random places before. So that is the net-new supply that we should go after. Everything else is going to look at lead times of 12 to 24 months. It's going to look pretty bad for a while, for building a 50-megawatt or 100-megawatt data center. So I do think economies of scale are actually on my side at the sub-megawatt scale.

And can I ask generally what lead times look like on that scale?

It seems reasonable. If you have power rights on site, I think you can get something up in 16 weeks.

Oh, that's very fast.

Really fast.

Wow. Okay. What if you scale from the one- to five-megawatt level?

It can strain your site selection a little bit. And if you're willing to still separate them into different pods, meaning it's not five megawatts in one contiguous cluster, but maybe five one-megawatt sites or five one-megawatt pods that are each independently networked to each other, that can be similar, not too much worse.

But if you want a contiguous five megawatts, which is about 2,000 or so chips, 2,500 chips, it's not easy. It can be done. But now you want a site that is qualified to do liquid cooling. You want a site that has some experience with— The larger the cluster, the more likely it is that you want to build it for training and just collect the training dollars on that, meaning you want high-reliability networking and so on.

And where are the step functions as far as getting credit or getting capital for that?

Credit is not easy either. Nobody wants to underwrite inference as a workload because no one knows if open source is going to be around tomorrow, or whether it's going to have the economics that we need to continue to monetize the GPUs the way we are today. It is very hard for creditors to underwrite these things. They really want to see a Microsoft offtake. They want to see investment-grade, IG offtake. And that's not really going to happen in the inference market, I think.

Neoclouds are not going to sell to Microsoft. They're going to sell to neolabs. They're going to sell to small operators. And so our view is that we're going to want to work with small providers, but they're also the least creditworthy. And the alchemy we can do there is try to be the trust arbiter. We can go do our own diligence and really be rigorous about saying, "Here is a site that we have vetted. We understand the constraints around power and cooling that the site has to provide. Here's why we're confident this is going to work."

And we happen to have expertise on the team that understands those kinds of constraints. Then we go to NVIDIA, who is also a great friend and supporter of the company, and we ask them, "Let's cooperate on a pod design that's going to work around the constraints of this site." And their site development team is the best in the game. NVIDIA has an extraordinarily good ability to understand what's going to delay or block a site coming online, and they can help us get ahead of those issues.

And so we work very closely with them on these kinds of things. And then finally, we quarterback the execution all the way through. We will work with the node provider, the OEM, like Supermicro or Dell, to build the servers. We can work with the site team to prepare everything in advance so that we're ready to rack the systems as soon as they go live.

You want someone with expertise across the use case, which is either training or inference, and all the way up to the hardware, to understand when Supermicro tells you that, "Hey, the memory is delayed. We have some constraint getting the memory, or the memory prices have gone up since we last quoted you." What do you do about it? Well, we have the bravery to say that maybe memory doesn't matter as much for us. The DDR5 in the CPUs, we can compromise on. Yes, go crazy. But we know what makes inference work. And we are ready to understand and optimize the entire system, including the spec of the hardware, including the rack topology for our use case.

This makes a lot of sense if you own and build out your own data centers, small size or not. This is a little bit harder if you work with other providers, right? So I'm curious about what that relationship is generally shaped like. How much access do they give you? How far can you debug? Or how much visibility do you have over their hardware?

Yes, great question. Providers are not really used to working with researchers or working with people who do machine learning workloads. They're used to operating hardware at high reliability. And so our job is to be the go-between between the person who is writing models or optimizing models, and the person who is physically in the data center racking these machines, checking for failures and responding to failures.

And so we try to buy bare-metal clusters whenever possible. We want to have the cleanest visibility into the machines. We are very self-service in our ability to debug issues with the machines. I've seen a lot of pain when a person who's training a model says their model is breaking, or they're just not able to make progress in their model, and they call the data center and say, "Hey, my PyTorch code doesn't work."

The data center operator has no clue how to fix the PyTorch code, and why should they? It's not their wheelhouse, not their expertise. And so it's really valuable to have someone like us sitting there and saying, "Well, we've seen this before." We know how to self-diagnose. We know how to interpret the Xids that NVIDIA's GPUs are throwing. We know how to escalate to NVIDIA if we think there's a fundamental issue with this chip.

We can tell the difference between HBM ECC failures, GPU core failures, or simply power issues in the data center, which, by the way, are increasingly a problem. There's so much stress placed on power systems these days, where training especially has frequent extreme power spikes, where you're going from all the GPUs being more or less idle when they're waiting for an all-reduce to finish, to a spike as they all get their data and start a large matmul right after that.

And so you have this extreme power transient where the GPUs go from drawing hundreds of watts to thousands of watts all at once. And NVIDIA has actually corrected for this in the latest designs of GB300, where they have supercapacitor trays in the rack to absorb the power spike locally. But some of the older machines and B300s don't always have that deployed. And so you can easily brown out the whole data center. You can knock it all out if you synchronize the workload across too many machines at once.

Do you see those spikes with inference as well?

Inference is far less likely to do that, partially because the atomic unit is smaller. We are not going to use 128 machines and nodes for inference. We're going to use more like eight machines—a topology that we frequently deploy—64 GPUs. It's much less likely that we have extreme synchronization across all the GPUs. It's all about the transient, the slope of the current draw.

And it sounds like you require full access, then, from your providers.

Absolutely. We need to be able to understand what's going on with the machines ourselves. It's really important that we have our theory of the world and what's happening, and we are really transparent with the provider. We don't think an adversarial relationship is a good thing to have. We want to be collaborative here. We want to understand what we can about the machines and then help the provider get to a resolution.

The reality is we're all flying blind here in some sense. This is very new: to rack hundreds of kilowatts of machines per rack and then have cooling that is being redesigned on the fly to match the needs of these new chips. It's best to have open and honest communication with your provider. That's something that we try to specialize in.

Does that limit the amount of compute that you can bring online? It depends on where your compute is coming from, but some providers are going to be cagier about this than others.

They will be. But the ones who are good, the ones that you want to build a relationship with, are not. They want to improve as well. They are, in fact, grateful to have a partner who understands what their constraints are and is willing to work with them to improve the quality of the cluster.

They're going to improve, and the next time they build the cluster, they can maybe try to go market it to OpenAI or Anthropic. They all want the biggest contracts. We want to work with new providers who are eager to learn and improve. If they work with someone bigger in the future, that's great. We have created value in the meantime, and we are very happy to continue working with sites that are fast to come online.

What do your agreements for compute typically look like? For instance, do they tend to be much shorter contracts? Are they more immediate contracts? Often you don't know if you're going to be able to lease out a cluster until very soon before it's available. I'm curious how that affects the shape of your relationships.

There's not really a short-contract option in the space anymore. I think that fundamentally all GPUs are financed by someone. Whether it's you or your data-center operator is maybe up in the air, but at the end of the day, the underlying contract is almost always three, four, or five years. We're seeing a trend up to four or five because people want to know that the machines are going to have a lifetime beyond just the initial term.

It's true that the payback period for these machines is frequently between the one-year and two-year mark for the operator. But there's enough risk in the ability to support those rates for three years that people charge super-high premiums to actually constrain the term just to that period. The underlying lease or depreciation cycle is usually five years or more. The closer you can get to that underlying lease term, the more likely it is that you're getting the pass-through credit terms from the upstream lender.

Anything shorter than that, somebody is advancing you some risk most of the time. We don't love that, obviously, and we want to be more nimble. We would rather have shorter terms. In some sense, we're in the business of predicting or underwriting what we think the future value of compute is going to be.

There are two schools of thought on this. One is that compute is going to continue to be scarce for the next three years, and therefore we don't see demand going down as an aggregate. You should just buy all the compute you can. That's one view of the world. The other is that there could be supply shocks as new chips come to market. There could be inference supply shocks in particular, which is, I think, the form of supply shock that even could happen.

Training, I don't think, is likely to move away from NVIDIA anytime soon, because people who train models are conservative. It's very much like the "no one gets fired for buying IBM" in the modern era. A training job either works or it doesn't. If you have any increase in the chance of the run not working for any reason—because the kernels weren't ready or the networking was bad or whatever—that will cause some people to lose everything on the training side.

Even if the compute could be half the cost if you use a different vendor, it's unlikely to move away from NVIDIA because of the increased risk, or the fear of that risk. It's the intolerance of variance, actually. You know that if you buy an NVIDIA cluster and you train the model, it's going to work because everyone's doing it all the time. But if you buy an alternative vendor's cluster and you try to train the model, there's less evidence that it will work. People are not trying to rock the boat. You want to spend your innovation tokens as a startup somewhere else. You don't want to risk infrastructure for training.

For inference, it's a whole different game. You know exactly when you run a forward pass whether it's going to work or not. There's much less bogeyman ambiguity. We think it's more practical to say, "Hey, we are an inference company. We're going to specialize in alternative hardware. We are going to understand our machines so well that we are going to bring that confidence ourselves," versus someone training a model who is typically interested in solving some other problem, and the infrastructure is just a background concern for them.

And as far as inference supply shocks, do you think that's mostly going to be, if that happens, due to alternative hardware?

Yeah. I think there are meaningfully different choices one can make for inference that are worth taking a risk on, at least. The challenge for the inference-chip companies is going to be whether they can execute. Remember, it's not just making a chip anymore; it's making a rack-scale system. Rack-scale systems are something that NVIDIA has pretty exclusive experience on for the last three years.

Also remember that the first rack-scale systems from NVIDIA were very painful. GB200 was not an easy rack-scale system to deploy. NVIDIA learned a lot from that experience, and the issues are fixed. But everyone else building these rack-scale systems—hopefully they can learn the lessons that NVIDIA did quickly.

All right, last question. Let's say five years from now, everything Sail has been trying to do pans out. How different is the world, and what kind of workloads are you guys running?

We've always predicted that async inference is going to eat the world. We think async inference is going to be the future of how agents run. They're going to run proactively, without waiting on human input, ahead of user demand at all times. We think that could easily be 95% or more of inference demand in total.

What Sail does in that world is we are serving all these tokens as maximally efficiently as possible. We're following the sun to make sure that we are running inference in places of the world where the grid is currently negative-priced because the solar power is so plentiful. We have distributed compute that is designed for efficiency and avoids the bottlenecks that we think are going to continue for quite a while on the memory side.

For example, we're going to be using older process nodes to redesign chips. Yes, we'll compromise something on performance per watt, but we will be using all the ideas that we have from modern AI ASICs and backporting them to older process nodes in order to get the lowest cost per FLOP. We'll use cheap, plentiful power to run those. We'll be running a great distributed-systems company.

That sounds beautiful. Well, thank you so much for taking the time. It was great to have the conversation.

Yes. Thank you so much for having me again. It was a great conversation.