Blog · · 7 minute read
We designed a core for Inkling 2 Small
Thinking Machines opened the weights of Inkling 2 Small. The checkpoint is 393 GB, and Thinking Machines trained the model in NVFP4 from start to finish. We used it to answer our most important question. What does silicon built for this model deliver? The answer and the study are open.
3–4x
faster for one user than B300 GPUs
Modeled
~4x
the tokens per joule at a busy, interactive operating point
Modeled
1.0x
decode cost against GPUs at 100 tokens a second per user: equal
Modeled, median
1.6M
vectors through our NVFP4 dot product, 0 mismatches
Simulated RTL
01 · The result
A core for fast tokens at scale
We modeled a core that keeps every weight of Inkling 2 Small in reloadable memory on the die. We compared it with B300 GPUs that run the best published serving engines of today. The GPUs get the layout that is best for them: 8x B300 for one user, 4x B300 when busy.
- Speed for one user
- 3–4x faster
- 2,960 against 720 tokens a second · range 1.3x to 8.2x
- Tokens per joule, busy and interactive
- About 4x
- range 2.0x to 8.2x
- Decode cost at 100 tokens a second per user
- Equal
- median 1.00x · range 0.31x to 2.63x
- Decode cost at 200 tokens a second per user
- About 4x lower
- where the GPU model is least certain
The faster each user needs tokens, the larger the gain.
Speed for one user
Tokens a second, 8K context, the model's draft on
02 · The baseline
Why a GPU cannot go faster for one user
A GPU keeps the weights in HBM and the arithmetic on the die. Thus the weights for each token must move from HBM to the die. For Inkling 2 Small, that is 19.3 GB per token. It is about 73 GB per step when the model's draft verifies eight tokens at a time. Today's engines also issue hundreds of dependent kernel launches per step. Arithmetic is 1% of the time.
Software alone recovers a part of that time. A compiled dataflow schedule on the same four GPUs gives 1.7x for one user. We count it. More speed than that needs different hardware.
03 · The design
Weights that never move
The core keeps all 393 GB of the checkpoint in SRAM across 521 dies. Each bank is adjacent to the multipliers that use it. The state of the token moves from layer to layer, and the weights stay where they are. The eight full-attention layers share one attention tile, with the KV cache in HBM.
The weights are reloadable. A new checkpoint, a fine-tune or the adapter of a customer is a load, not a tape-out. Weights fixed in silicon need 95 new die designs each time the model changes.
Customizable models need customizable silicon.
04 · The silicon
We built the part that does the math
The experts of Inkling 2 Small are NVFP4: 4-bit values with a shared 8-bit scale per block of 16. We wrote the block-scaled dot product in SystemVerilog and generated it from the model's own config. We checked it bit for bit against a reference, on real expert weights from the checkpoint. The result was 1,589,011 vectors, 0 mismatches. The result of a request does not depend on the other requests in the batch. We placed and routed the design on an open 7 nm kit, and it closed timing at 556 MHz.
It was larger than we assumed, and we put the real numbers into the model. It also showed that a fixed multiplier weight saves only 13% of the multiplier area. Reloadable arithmetic is cheap. The real cost of reloadability is memory.
05 · The economics
Where the gains are
On the agentic workload that the serving engines publish, prompts are 34% of the GPU cost per output token. When there are several prompts at the same time, the core also handles them.
- Weights on-die for decode and for prefill
- 1.4x
- lower cost per output token than GPUs · 3.3x the tokens per joule
- Weights on-die for decode, weights in HBM for prefill
- 1.9x
- lower cost · 3.0x the tokens per joule
- Weights in HBM, for both
- 2.2x
- lower cost · the cheapest whole system we modeled, and no faster for one user
Decode cost, GPUs divided by the core
At the speed promised to each user. Above 1x, the core is cheaper.
Fast tokens at scale.
At 100 tokens a second per user, the decode cost of the core is equal to that of GPUs. At 200 tokens a second, each GPU serves only a few users, and the core is about 4.3x cheaper.
Energy.
The core gives about 4x the tokens per joule at a busy, interactive operating point.
Scale.
It is a large machine of 521 dies. It recovers its design cost above about 1.6 trillion tokens a day of one model.
Smaller models, smaller gain.
For smaller models with the same shape as this one, the speed gain decreases, to 2.8x at about 8 billion parameters. We predicted the opposite.
06 · The method
We tried to prove it wrong
We wrote down 36 numeric predictions before we ran a simulation. Of these, 21 were in the predicted range, and we list each miss with its reason.
Then two new reviewers looked for faults in the study and found 23 problems. For example, the GPU fit did not include prefill load. There was a GPU layout that we did not try. Our prefill model left most of our own core idle. We corrected each problem that we were able to verify. Our first cost estimate decreased from 9.5x to 1.8x, and speed decreased from 7.4x to 4.1x.
The numbers on this page are the ones that passed these checks.
07 · Next
Everything is open
The simulator, the calibration, the RTL, the predictions, both review logs and every figure regenerate with one command.
Two measurements will improve the study more than any result that we can model. The first is an expert-routing histogram from real traffic. The second is a GPU power reading while the GPU serves one request. If you have one of them, tell us. If you run your own model and want to know what a core can do for it, tell us what you run.
Notes
This is independent work by Aercrist. It is not affiliated with or endorsed by Thinking Machines Lab. The core figures are modeled, and we calibrated the GPU figures to measurements that vLLM and TokenSpeed published. We give speed as a range from the median of our uncertainty analysis to its central case. Each range is the 5th to 95th percentile. A script generates each number on this page from the results of the study at commit 4f992be.
