Blog · · 7 minute read

We designed a core for Inkling 2 Small

Thinking Machines opened the weights of Inkling 2 Small. The checkpoint is 393 GB, and Thinking Machines trained the model in NVFP4 from start to finish. We used it to answer our most important question. What does silicon built for this model deliver? The answer and the study are open.

  • 3–4x

    faster for one user than B300 GPUs

    Modeled

  • ~4x

    the tokens per joule at a busy, interactive operating point

    Modeled

  • 1.0x

    decode cost against GPUs at 100 tokens a second per user: equal

    Modeled, median

  • 1.6M

    vectors through our NVFP4 dot product, 0 mismatches

    Simulated RTL

01 · The result

A core for fast tokens at scale

We modeled a core that keeps every weight of Inkling 2 Small in reloadable memory on the die. We compared it with B300 GPUs that run the best published serving engines of today. The GPUs get the layout that is best for them: 8x B300 for one user, 4x B300 when busy.

Speed for one user
3–4x faster
2,960 against 720 tokens a second · range 1.3x to 8.2x
Tokens per joule, busy and interactive
About 4x
range 2.0x to 8.2x
Decode cost at 100 tokens a second per user
Equal
median 1.00x · range 0.31x to 2.63x
Decode cost at 200 tokens a second per user
About 4x lower
where the GPU model is least certain

The faster each user needs tokens, the larger the gain.

Speed for one user

Tokens a second, 8K context, the model's draft on

  • B300 GPUs2x, today's engines380 tok/s
  • B300 GPUs4x, today's engines550 tok/s
  • B300 GPUs8x, today's engines720 tok/s
  • B300 GPUs4x, compiled software930 tok/s
  • Core, weights in HBM4 packages790 tok/s
  • Core, weights on-die521 dies, reloadable2,960 tok/s
Modeled. We calibrated the GPU rows to 9 figures that vLLM and TokenSpeed published, to 8% RMS.

02 · The baseline

Why a GPU cannot go faster for one user

A GPU keeps the weights in HBM and the arithmetic on the die. Thus the weights for each token must move from HBM to the die. For Inkling 2 Small, that is 19.3 GB per token. It is about 73 GB per step when the model's draft verifies eight tokens at a time. Today's engines also issue hundreds of dependent kernel launches per step. Arithmetic is 1% of the time.

Software alone recovers a part of that time. A compiled dataflow schedule on the same four GPUs gives 1.7x for one user. We count it. More speed than that needs different hardware.

03 · The design

Weights that never move

The core keeps all 393 GB of the checkpoint in SRAM across 521 dies. Each bank is adjacent to the multipliers that use it. The state of the token moves from layer to layer, and the weights stay where they are. The eight full-attention layers share one attention tile, with the KV cache in HBM.

The weights are reloadable. A new checkpoint, a fine-tune or the adapter of a customer is a load, not a tape-out. Weights fixed in silicon need 95 new die designs each time the model changes.

Customizable models need customizable silicon.

04 · The silicon

We built the part that does the math

The experts of Inkling 2 Small are NVFP4: 4-bit values with a shared 8-bit scale per block of 16. We wrote the block-scaled dot product in SystemVerilog and generated it from the model's own config. We checked it bit for bit against a reference, on real expert weights from the checkpoint. The result was 1,589,011 vectors, 0 mismatches. The result of a request does not depend on the other requests in the batch. We placed and routed the design on an open 7 nm kit, and it closed timing at 556 MHz.

It was larger than we assumed, and we put the real numbers into the model. It also showed that a fixed multiplier weight saves only 13% of the multiplier area. Reloadable arithmetic is cheap. The real cost of reloadability is memory.

05 · The economics

Where the gains are

On the agentic workload that the serving engines publish, prompts are 34% of the GPU cost per output token. When there are several prompts at the same time, the core also handles them.

Weights on-die for decode and for prefill
1.4x
lower cost per output token than GPUs · 3.3x the tokens per joule
Weights on-die for decode, weights in HBM for prefill
1.9x
lower cost · 3.0x the tokens per joule
Weights in HBM, for both
2.2x
lower cost · the cheapest whole system we modeled, and no faster for one user

Decode cost, GPUs divided by the core

At the speed promised to each user. Above 1x, the core is cheaper.

  • 25 tok/sper user1.0x
  • 50 tok/sper user1.2x
  • 100 tok/sper user1.8x
  • 150 tok/sper user3.1x
  • 200 tok/sper user4.3x
Modeled, at 32K context. The GPU model is least certain at the top speeds, where each GPU serves only a few users.
  • Fast tokens at scale.

    At 100 tokens a second per user, the decode cost of the core is equal to that of GPUs. At 200 tokens a second, each GPU serves only a few users, and the core is about 4.3x cheaper.

  • Energy.

    The core gives about 4x the tokens per joule at a busy, interactive operating point.

  • Scale.

    It is a large machine of 521 dies. It recovers its design cost above about 1.6 trillion tokens a day of one model.

  • Smaller models, smaller gain.

    For smaller models with the same shape as this one, the speed gain decreases, to 2.8x at about 8 billion parameters. We predicted the opposite.

06 · The method

We tried to prove it wrong

We wrote down 36 numeric predictions before we ran a simulation. Of these, 21 were in the predicted range, and we list each miss with its reason.

Then two new reviewers looked for faults in the study and found 23 problems. For example, the GPU fit did not include prefill load. There was a GPU layout that we did not try. Our prefill model left most of our own core idle. We corrected each problem that we were able to verify. Our first cost estimate decreased from 9.5x to 1.8x, and speed decreased from 7.4x to 4.1x.

The numbers on this page are the ones that passed these checks.

07 · Next

Everything is open

The simulator, the calibration, the RTL, the predictions, both review logs and every figure regenerate with one command.

Two measurements will improve the study more than any result that we can model. The first is an expert-routing histogram from real traffic. The second is a GPU power reading while the GPU serves one request. If you have one of them, tell us. If you run your own model and want to know what a core can do for it, tell us what you run.

Tell us what you runRead the study

Notes

This is independent work by Aercrist. It is not affiliated with or endorsed by Thinking Machines Lab. The core figures are modeled, and we calibrated the GPU figures to measurements that vLLM and TokenSpeed published. We give speed as a range from the median of our uncertainty analysis to its central case. Each range is the 5th to 95th percentile. A script generates each number on this page from the results of the study at commit 4f992be.