Blog · · 6 minute read

We designed a decode chip for DeepSeek-V4.1-Flash

Baseten's deepseek-ultrafast serves DeepSeek-V4.1-Flash, a 552B-parameter model, at 1,740 tokens a second for one user on four GB300 GPUs. The engine is open, so we rebuilt its workload from the source. We asked two questions. Where does the rest of the time go? What does a chip built for this model deliver? The answers and the study are open.

  • 2.1x

    faster for one user than deepseek-ultrafast, measured on four GB300s

    Modeled

  • 1.8x

    the tokens per joule of GPUs that run the same software

    Modeled

  • 1.5x

    lower cost per million tokens than GPUs that run the same software

    Modeled

  • 256/256

    test cases from our router RTL match the reference model

    Simulated RTL

01 · The result

Twice the speed, for 44% less power

We modeled MDA-1, a decode chip for DeepSeek-V4.1-Flash. We compared it with the engine that Baseten measured and with the best software that we can project for the same GPUs. The comparison is four packages against four GB300s, for one user, with the model's draft on.

Speed for one user
2.1x faster
3,656 against a measured 1,740 tokens a second · range 1.6x to 2.3x
Tokens per joule
1.8x
against GPUs that run the same software · range 1.2x to 3.0x
Cost per million tokens
1.5x lower
against GPUs that run the same software · range 0.9x to 2.1x
The same chip with HBM4
6,159 tokens a second
3.5x the measured engine · 2.67 tokens per joule

Half of the step is wait time. Remove the wait time and the speed doubles.

Speed for one user

Tokens a second, 2.6K context, the model's draft on

  • 16x Alveo V80FPGA1,180 tok/s
  • 4x GB300deepseek-ultrafast, measured1,740 tok/s
  • 8x B300deepseek-ultrafast, measured2,060 tok/s
  • 4x GB300dataflow software3,596 tok/s
  • 4x MDA-1decode chip3,656 tok/s
The two deepseek-ultrafast rows are Baseten's measurements. The other rows are modeled and calibrated to those two.

02 · The baseline

Where the time of a decode step goes

A step takes 3.30 ms on four GB300s. Our model puts 1.85 ms of it, 56%, into fixed latency. A step has 388 dependent kernel stages, and each stage exposes about 4.2 µs. The all-reduces between GPUs add to the fixed latency. The memory that supplies the weights averages 37% of its peak bandwidth. For most of the step, the GPUs wait for each other.

Thus more bandwidth alone does not solve the problem. Rubin has 2.4x the memory bandwidth of GB300. With today's per-stage latency, that bandwidth gives 35% more speed. The wait time must go first.

03 · The execution model

One program that never stops

The solution is an execution model. One persistent program in a fixed order replaces hundreds of dependent kernel launches. Flags do the hand-offs between stages. The program streams the weights of the next layers while the experts of the current layer still run.

On the same four GB300s, the model projects 3,596 tokens a second for one user. That is 2.1x the measured engine. The persistent program adds 61%, flags add 7% and prefetch adds 23%. The execution model needs no new hardware. We count it, and we compare our chip with it.

The GPUs that are in the rack now can give 2x.

Where the speed comes from

Tokens a second for one user, each step added to the step before

  • Todaycalibrated model1,697 tok/s
  • One persistent programsoftware2,736 tok/s
  • Flag hand-offssoftware2,919 tok/s
  • Weight prefetchsoftware3,596 tok/s
  • MDA-1silicon3,656 tok/s
Modeled, four chips. Every step before the last runs on four GB300s.

04 · The design

A chip built for that program

MDA-1 is that execution model in silicon. Each package has two dies and eight HBM3E stacks at 8 TB/s, the same bandwidth as a GB300. Each package also has 2 PF of FP8 arithmetic, a hardware MoE router and an all-reduce in the fabric. It runs at 94% of what its memory can supply.

It does one job. Prompts stay on a GPU pool and the chip decodes. For an agent turn, the pool hands over 5 MB of KV cache in 0.12 ms. The weights are in HBM, so a new checkpoint is a load, not a tape-out.

Its speed matches the software: 3,656 tokens a second, 1.02x the GPUs that run the same program. It uses 1.87 kW, and the GPUs use 3.36 kW.

Software sets the speed. Silicon sets the cost of the speed.

05 · The silicon

We built the router

Each token picks 6 of 384 experts in each layer, and no later stage can start before that. We wrote that top-k router in SystemVerilog and checked it against a reference model. 256/256 test cases match. The router takes 98 cycles, which is 65 ns at 1.5 GHz.

We also ran the matrix engine in SCALE-Sim, a cycle-level simulator. It showed that our efficiency assumption was 1.6x optimistic, so we decreased it. The one-user numbers did not change. At batch 1, memory limits the chip, not arithmetic.

06 · The economics

Where the gains are

A GPU is built to do all tasks, and decode uses a small part of it. MDA-1 has the memory bandwidth of a GB300 and only the arithmetic that decode needs. Thus the same tokens need less power and less money.

MDA-1 also matches the next generation. Rubin with the new software reaches 6,149 tokens a second, 2.7x the speed that Rubin gets from today's software. MDA-1 with the same HBM4 reaches 6,159.

Decode cost per million tokens

Owned hardware for four years, one user per system. A shorter bar is cheaper.

  • 4x GB300today's engine$0.91
  • 4x GB300dataflow software$0.43
  • 4x MDA-1HBM3E$0.27
  • 4x MDA-1HBM4$0.19
Modeled. GPUs at $45,000 each and 60% of rated power. MDA-1 at manufacturing cost plus $300M of design cost divided across 20,000 chips.
  • Energy.

    Four MDA-1 packages use 1.87 kW and four GB300s use 3.36 kW for the same speed. That is 1.8x the tokens per joule.

  • Cost.

    The cost per million tokens is 1.5x lower than for GPUs that run the same software. The chip is cheaper at every batch size we simulated, from 1 to 256.

  • Scale.

    A $300M design recovers its cost at about 9,000 chips against GPUs that run the new software. Against today's engine, it recovers its cost at 3,400 chips.

  • The next memory.

    A respin of the chip with HBM4 reaches 6,159 tokens a second for one user, at 2.67 tokens per joule.

07 · The method

We tried to prove it wrong

We read the workload from the Rust and CUDA source of the engine and from the 48 checkpoint headers of the model. The headers reproduce the published parameter counts. We calibrated the simulator to the two step times that Baseten measured. We refit it with 10 alternative assumptions, and we drew all chip constants jointly 300 times.

Then a reviewer examined the study two times and looked for the objections that experts can make. The reviewer raised 14 points, and we addressed each one. In our first draft, the chip was faster than the software. With the same physics on the two sides, the speeds match, and the case for silicon became cost and energy. We also ran the alternatives on the same simulator. Sixteen FPGAs reach 1,180 tokens a second, and an on-chip expert cache needs gigabytes to help.

The numbers on this page are the ones that passed these checks.

08 · Next

Everything is open

The simulator, the calibration, the router RTL, both rounds of review and every figure regenerate with one command.

One measurement will improve the study more than any result that we can model: one profile of the engine on a GPU. The study includes a tool that prints our prediction in the profile groups of the engine, ready for comparison. If you have that profile, tell us. If you run your own model and want to know what a chip can do for it, tell us what you run.

Tell us what you runRead the study

Notes

This is independent work by Aercrist. It is not affiliated with or endorsed by Baseten or DeepSeek. The MDA-1 figures and the GPU-software figures are modeled. No chip exists. The measured figures come from the deepseek-ultrafast README, and we calibrated the GPU model to them.

Across our refits, fixed latency is 43% to 62% of a step. Speed ranges are the 5th to 95th percentile of our uncertainty analysis. Cost and energy ranges cover GPU prices of $25,000 to $65,000 and 40% to 100% of rated power. Each number on this page comes from the results of the study at commit d2e9d7f.