Blog · · 6 minute read
We designed a decode chip for DeepSeek-V4.1-Flash
Baseten's deepseek-ultrafast serves DeepSeek-V4.1-Flash, a 552B-parameter model, at 1,740 tokens a second for one user on four GB300 GPUs. The engine is open, so we rebuilt its workload from the source. We asked two questions. Where does the rest of the time go? What does a chip built for this model deliver? The answers and the study are open.
2.1x
faster for one user than deepseek-ultrafast, measured on four GB300s
Modeled
1.8x
the tokens per joule of GPUs that run the same software
Modeled
1.5x
lower cost per million tokens than GPUs that run the same software
Modeled
256/256
test cases from our router RTL match the reference model
Simulated RTL
01 · The result
Twice the speed, for 44% less power
We modeled MDA-1, a decode chip for DeepSeek-V4.1-Flash. We compared it with the engine that Baseten measured and with the best software that we can project for the same GPUs. The comparison is four packages against four GB300s, for one user, with the model's draft on.
- Speed for one user
- 2.1x faster
- 3,656 against a measured 1,740 tokens a second · range 1.6x to 2.3x
- Tokens per joule
- 1.8x
- against GPUs that run the same software · range 1.2x to 3.0x
- Cost per million tokens
- 1.5x lower
- against GPUs that run the same software · range 0.9x to 2.1x
- The same chip with HBM4
- 6,159 tokens a second
- 3.5x the measured engine · 2.67 tokens per joule
Half of the step is wait time. Remove the wait time and the speed doubles.
Speed for one user
Tokens a second, 2.6K context, the model's draft on
02 · The baseline
Where the time of a decode step goes
A step takes 3.30 ms on four GB300s. Our model puts 1.85 ms of it, 56%, into fixed latency. A step has 388 dependent kernel stages, and each stage exposes about 4.2 µs. The all-reduces between GPUs add to the fixed latency. The memory that supplies the weights averages 37% of its peak bandwidth. For most of the step, the GPUs wait for each other.
Thus more bandwidth alone does not solve the problem. Rubin has 2.4x the memory bandwidth of GB300. With today's per-stage latency, that bandwidth gives 35% more speed. The wait time must go first.
03 · The execution model
One program that never stops
The solution is an execution model. One persistent program in a fixed order replaces hundreds of dependent kernel launches. Flags do the hand-offs between stages. The program streams the weights of the next layers while the experts of the current layer still run.
On the same four GB300s, the model projects 3,596 tokens a second for one user. That is 2.1x the measured engine. The persistent program adds 61%, flags add 7% and prefetch adds 23%. The execution model needs no new hardware. We count it, and we compare our chip with it.
The GPUs that are in the rack now can give 2x.
Where the speed comes from
Tokens a second for one user, each step added to the step before
04 · The design
A chip built for that program
MDA-1 is that execution model in silicon. Each package has two dies and eight HBM3E stacks at 8 TB/s, the same bandwidth as a GB300. Each package also has 2 PF of FP8 arithmetic, a hardware MoE router and an all-reduce in the fabric. It runs at 94% of what its memory can supply.
It does one job. Prompts stay on a GPU pool and the chip decodes. For an agent turn, the pool hands over 5 MB of KV cache in 0.12 ms. The weights are in HBM, so a new checkpoint is a load, not a tape-out.
Its speed matches the software: 3,656 tokens a second, 1.02x the GPUs that run the same program. It uses 1.87 kW, and the GPUs use 3.36 kW.
Software sets the speed. Silicon sets the cost of the speed.
05 · The silicon
We built the router
Each token picks 6 of 384 experts in each layer, and no later stage can start before that. We wrote that top-k router in SystemVerilog and checked it against a reference model. 256/256 test cases match. The router takes 98 cycles, which is 65 ns at 1.5 GHz.
We also ran the matrix engine in SCALE-Sim, a cycle-level simulator. It showed that our efficiency assumption was 1.6x optimistic, so we decreased it. The one-user numbers did not change. At batch 1, memory limits the chip, not arithmetic.
06 · The economics
Where the gains are
A GPU is built to do all tasks, and decode uses a small part of it. MDA-1 has the memory bandwidth of a GB300 and only the arithmetic that decode needs. Thus the same tokens need less power and less money.
MDA-1 also matches the next generation. Rubin with the new software reaches 6,149 tokens a second, 2.7x the speed that Rubin gets from today's software. MDA-1 with the same HBM4 reaches 6,159.
Decode cost per million tokens
Owned hardware for four years, one user per system. A shorter bar is cheaper.
Energy.
Four MDA-1 packages use 1.87 kW and four GB300s use 3.36 kW for the same speed. That is 1.8x the tokens per joule.
Cost.
The cost per million tokens is 1.5x lower than for GPUs that run the same software. The chip is cheaper at every batch size we simulated, from 1 to 256.
Scale.
A $300M design recovers its cost at about 9,000 chips against GPUs that run the new software. Against today's engine, it recovers its cost at 3,400 chips.
The next memory.
A respin of the chip with HBM4 reaches 6,159 tokens a second for one user, at 2.67 tokens per joule.
07 · The method
We tried to prove it wrong
We read the workload from the Rust and CUDA source of the engine and from the 48 checkpoint headers of the model. The headers reproduce the published parameter counts. We calibrated the simulator to the two step times that Baseten measured. We refit it with 10 alternative assumptions, and we drew all chip constants jointly 300 times.
Then a reviewer examined the study two times and looked for the objections that experts can make. The reviewer raised 14 points, and we addressed each one. In our first draft, the chip was faster than the software. With the same physics on the two sides, the speeds match, and the case for silicon became cost and energy. We also ran the alternatives on the same simulator. Sixteen FPGAs reach 1,180 tokens a second, and an on-chip expert cache needs gigabytes to help.
The numbers on this page are the ones that passed these checks.
08 · Next
Everything is open
The simulator, the calibration, the router RTL, both rounds of review and every figure regenerate with one command.
One measurement will improve the study more than any result that we can model: one profile of the engine on a GPU. The study includes a tool that prints our prediction in the profile groups of the engine, ready for comparison. If you have that profile, tell us. If you run your own model and want to know what a chip can do for it, tell us what you run.
Notes
This is independent work by Aercrist. It is not affiliated with or endorsed by Baseten or DeepSeek. The MDA-1 figures and the GPU-software figures are modeled. No chip exists. The measured figures come from the deepseek-ultrafast README, and we calibrated the GPU model to them.
Across our refits, fixed latency is 43% to 62% of a step. Speed ranges are the 5th to 95th percentile of our uncertainty analysis. Cost and energy ranges cover GPU prices of $25,000 to $65,000 and 40% to 100% of rated power. Each number on this page comes from the results of the study at commit d2e9d7f.
