Jin Daily AI Trivia: NVIDIA Nemotron Diffusion Model is SO FAST
NVIDIA’s Nemotron-Labs-TwoTower is a block-wise autoregressive diffusion language model built on top of the Nemotron-3-Nano-30B-A3B backbone.
Normal LLMs generate text one token at a time.
But TwoTower does something different.
It generates text by iteratively denoising a block of tokens in parallel, instead of decoding every token one by one.
So how is this possible?
Nemotron 3 Nano uses a hybrid Mamba-2 + Transformer + MoE design. TwoTower then splits the job into two towers:
One is the AR Context Tower. This frozen tower reads the prompt and committed tokens normally, then produces the KV cache and Mamba states.
The other is the Diffusion / Denoiser Tower. This tower takes a noisy block of tokens and tries to clean it up in parallel.
At the default setting, it uses a block size of 16 tokens. It does not magically complete every 16 tokens in one single step, but it can predict the whole block in parallel and commit the high-confidence tokens much faster than normal autoregressive decoding.
The result?
NVIDIA claims around 2.42x higher wall-clock generation throughput, while still retaining 98.7% of the AR baseline benchmark quality.
So if normal Nemotron 3 Nano is around 130-150 tokens/sec, then TwoTower can be roughly around 310-360 tokens/sec, depending on the setup.
But of course, there is a catch.
Full TwoTower diffusion inference uses 2 GPUs, with around 59GB per GPU for BF16 weights.
So yes, it is fast.
But also very NVIDIA:
“The more you buy, the more you save (time)!!”
