Nvidia paid $20 billion for speed. Did free software get there first?

The theoretical ceiling on how fast an eight-GPU Nvidia B200 server can serve one user is 3,047 tokens a second. That number comes from memory bandwidth alone, and real inference engines come nowhere near it.

Bandwidth grew, latency did not

To compute anything a GPU launches a small unit of work called a kernel, tears it down when it finishes, and launches the next one. Each launch and teardown takes a very short time. The trouble is that once the time to produce a single piece of the answer drops below a millisecond, that very short time becomes a large share of the whole. SemiAnalysis adds one line here. GPU memory bandwidth has grown roughly two to three times per generation while memory latency has not improved at all. A wider road does not shorten the trip if the starting gun keeps going off late.

Launch the kernel only once

What TileRT does is compile the whole answer-generating path ahead of time and freeze it into a single kernel. With the launching and tearing down gone, computation, memory reads and communication overlap each other. On a single B200 server it produced 500 tokens a second, which SemiAnalysis reports is about three times faster than a much larger GB300 NVL72 running conventional engines. On an equal-cost basis the gain is up to twofold. The software comes from the team behind the TileLang language and sits on GitHub under an MIT license. Xiaomi and Z.ai already run it behind the high-speed modes of their own services.

The $20 billion Nvidia paid Groq

Why speed carries money is something a price list answers.

Back in March The Register laid the structure out this way. Profit in the inference business comes from a trade between how many users one machine can hold and how fast it answers any one of them, and both cannot be maximised at once. Until now the thing that changed the terms of that trade was purpose-built silicon.

Nvidia confirms $20 billion Groq deal (Tom's Hardware, 2025-12-29)

출처: tomshardware.com

Nvidia said in December that it had signed a non-exclusive technology licensing agreement with Groq, reportedly worth $20 billion. Cerebras listed on Nasdaq in May at an offer price of $185 and posted core revenue of $191.3 million in the first quarter. That is the price the market has put on the ability to cut latency. So the question this piece leaves is not which side wins, hardware or software, but how much of a gap is left. If free software on GPUs closes in on purpose-built chips, the premium those chips collect shrinks to whatever that gap is worth. Cerebras hitting its $855 million guidance this year would say the premium is still alive; falling short would be the first evidence that software took the seat first.