The memory Kimi K3 saves does not arrive intact on real servers

A teardown of Kimi K3, the model whose weights Moonshot AI released on July 27, has landed. The reason it mixes two kinds of attention is not accuracy. It is memory.

Kimi K3 Is Here: Efficient Day-0 Support on vLLM (vLLM Blog, 2026-07-27)

출처: vllm-project.github.io

vLLM, the inference engine that actually serves this model, describes the design as "a linear-attention mechanism that keeps a fixed-size recurrent state instead of a growing KV cache."

What it means for the cache not to grow

A model that writes one word after another holds on to intermediate results so it does not recompute what it already read. That luggage is the KV cache. The longer the conversation runs, the heavier it gets, and the weight comes straight out of GPU memory. Linear attention compresses everything before it into a single fixed-size state. However long the text gets, the luggage stays the same size. Kimi K3 stacks that layer against a conventional layer that remembers everything, in a 3 to 1 ratio. Of roughly 2.8 trillion parameters only a fraction fire on any one token, and the model reads up to one million tokens at once.

Held down once every 32,000 tokens

Serving reality gets in the way here. To reuse the front of a conversation, the state at that point has to be sitting somewhere. A fixed-size state gets overwritten as it moves forward, so you have to leave markers to come back to. vLLM writes that it keeps checkpoints of the state at intervals, and the interval it gives as an example is 32,000 tokens. That value is a configurable default, not a hard-coded constant. Tighten the interval and memory goes up. Widen it and the next request recomputes the front of the prompt. So in real serving this model does not hold a constant amount of memory.

Being cheap and using less memory are two different things

Right after release, Kimi K3 came third on the Artificial Analysis intelligence index. In the same tally, the cost of running one task split like this.

Kimi K3 achieves #3 in the Artificial Analysis Intelligence Index (Artificial Analysis, 2026-07-17)

출처: artificialanalysis.ai

Artificial Analysis wrote that Kimi K3 scores comparably to Opus 4.8 and GPT-5.5, with only two models placed above it. The path to cheaper output is shifting away from buying more chips and toward producing the same answer with less memory. The saved memory does not simply become lower demand, though. A lighter cache means the same hardware now takes longer contexts and more simultaneous users, and the space that was freed fills back up.

The line Moonshot drew in its license

The weights are public, but not everyone is free to sell service on top of them. The developer Simon Willison pointed out that providers above 20 million dollars in annual revenue have to negotiate a separate commercial agreement with Moonshot. It is open to whoever runs it themselves, and half shut to whoever wants to resell it. Where the next model pushes this design is the fork to watch. If Moonshot strips out the full-attention layers in Kimi K4, it has bet the house on shrinking memory. If it puts more of those layers back, the gain from a smaller cache was worth less in production than it looked on paper.