Predicting 1 token at a time. That's what has been limiting models since the beginning, local ones included.
The constraint had nothing to do with hardware being insufficient or models being too small. It was architectural, baked in from the start.
To generate each token, your GPU loads all the model weights from memory, produces 1 token, then starts over. That memory bandwidth bottleneck is why local inference stayed frustrating even on decent hardware. And it's why 5 years of optimizations (flash attention, quantization, speculative decoding) all worked around the same structural ceiling without ever moving it.
TLDR: DiffusionGemma generates 256 tokens in parallel per denoising pass, shifting the bottleneck from memory bandwidth to raw compute. On paper: 4x faster than Gemma 4 AR, 700+ tokens/sec on RTX 5090, runs on RTX 4090. But the speed number is not what makes this interesting.
There's something that happens when you realize you've been optimizing the right metric on the wrong layer. Flash attention was a genuine breakthrough, INT4 quantization was a genuine breakthrough, and the H100 at $40,000 a card was a genuine breakthrough for anyone who could afford it. And none of it moved the fundamental constraint. The weights still had to load for every single token. The GPU's tensor cores, designed for massive parallel matrix operations, were mostly sitting idle during inference, just waiting for the next memory cycle. Every engineer who ran local inference on a high-end consumer GPU felt this as a kind of low-grade frustration: the numbers should work, they don't quite.
The Race Nobody Questioned
Flash attention dropped in 2022. Legitimately brilliant: rewriting the attention computation to be I/O-aware, keeping intermediate values in SRAM instead of constantly writing back to HBM. Real speedups, measurable, deployed everywhere within 18 months. The paper got 15,000 citations in roughly 2 years.
Then the field kept adding layers on the same foundation. INT4 quantization, GGUF, speculative decoding, Groq's LPU built from scratch around AR inference, H100s at $40,000 a card, then H200s, then GB200s. Entire companies valued in the billions on the premise that making AR faster was the problem worth solving. Groq built custom silicon from scratch around AR inference, and the industry called it visionary. Nobody in the room suggested that maybe the architecture itself was the question.
The whole industry organized around a single bottleneck. Nobody questioned whether the bottleneck was architectural.
To be fair: why would they? The models shipped, the products worked. The training stack, the inference serving infrastructure, the CUDA kernel ecosystem, every deployment pattern from vLLM to TGI to Ollama, all of it built around autoregressive next-token prediction. Questioning the architecture from inside that ecosystem is like questioning whether cars should have wheels while you're in the middle of designing a faster tire. The switching cost wasn't just technical. It was the industry's accumulated sunk cost: every CUDA kernel, every serving optimization, every hardware purchase justified against AR throughput numbers.
Inception Labs was the first to actually ship something different. Mercury came out early 2025, Mercury 2 in early 2026, 1,000+ tokens per second. Genuinely impressive numbers. Completely inaccessible: commercial API, closed weights, you couldn't run it yourself. Useful as a market signal, not actionable for anyone building on their own hardware.
DiffusionGemma shipped June 10, 2026, open weights under Apache 2.0, vLLM support on day 0, running on an RTX 4090.
For the developer community, "first open release from a tier-1 lab" means day-0 vLLM support, a HuggingFace model card, Unsloth integration, and a community that will have interesting fine-tunes out within days. The gap between a research paper and something you can actually build on closed roughly 18 months faster than anyone expected when Gemini Diffusion was announced as an experiment at I/O 2025.
This is the difference between "someone proved it works in theory" and "you can pull the weights tonight."
5 years of workarounds. The actual fix ran on your GPU yesterday.
What DiffusionGemma Actually Changes (and What It Doesn't)
The AR inference loop is memory-bound. To generate 256 tokens, your GPU loads the full weight matrix from memory 256 times, 1 load per token. The tensor cores, designed for parallel matrix multiplications at massive scale, execute their actual computation in roughly 1% of the total time. The other 99% is waiting for data to arrive from VRAM. Imagine staffing a kitchen with Michelin-starred chefs and routing every single plate through a warehouse 300 meters away between courses. That's your H100 on AR inference. This is why scaling a GPU's theoretical FLOPS rarely translated linearly to inference speed: you weren't compute-bound, you were memory-bound, and buying a card with more tensor cores helped less than buying one with higher memory bandwidth. Every optimization from flash attention onward was working on shortening that 99% wait, not eliminating it. The ceiling was always the same ceiling, just approached from a slightly different angle.
DiffusionGemma loads the weights once per denoising pass and generates 256 tokens in parallel. The bottleneck shifts. The tensor cores are now the actual ceiling, running bidirectional attention over the full 256-token block on each forward pass. This is what these chips were designed for. The memory bandwidth wall doesn't disappear, it stops being the thing that limits you.
Numbers from Google and NVIDIA: 700+ tokens/sec on RTX 5090, 1,000+ on H100, 4x faster than Gemma 4 AR on equivalent hardware. The model is 26B parameters as a Mixture of Experts, 3.8B active during inference, runs in 18GB VRAM when quantized.
The caveats are real and worth stating plainly.
DiffusionGemma trails Gemma 4 AR on reasoning benchmarks by a meaningful margin. AIME 2026: 69.1% vs 88.3%. LiveCodeBench v6: 69.1% vs 77.1%. GPQA Diamond: 73.2% vs 82.3%. These are 15-20 point gaps on hard reasoning tasks, not rounding errors.
The context window is 8,192 tokens. Most current AR models run at 128K+. For anything agentic or long-context, this is a real wall. A moderately complex TypeScript file eats 3,000 tokens. 3 files and you're already at the ceiling.
Google themselves call it "experimental." Fine-tuning recipes were still being published at launch. MLX support for Apple Silicon was incomplete day 0. For high-volume cloud serving, AR models still batch more efficiently at scale.
The performance is real, and so is the ceiling. The question is whether the ceiling matters for your use case.
The Proof Beyond Speed
A Sudoku board has 81 cells. Each one is constrained by its row, its column, and its 3x3 square simultaneously. To solve it correctly, you need to hold all those constraints in view at once.
An autoregressive model generates 1 cell at a time, left to right, top to bottom. By the time it fills cell 72, it cannot go back and correct cell 3. It conditions only on what it already generated. This isn't a failure of scale or a training data problem. It's a structural property of sequential generation. You can make an AR model bigger, faster, better at pattern matching, and it will still fill cells without global constraint resolution, because it structurally cannot look forward.
Google ran a test on this directly. DiffusionGemma base model on Sudoku puzzles: 0% success rate. Standard SFT fine-tuning with a JAX recipe on a Sudoku dataset: 80% success rate, with 4x fewer inference steps than the baseline.
The improvement came from bidirectional attention, not raw speed. Every token in the 256-token block attends to every other token during generation. The model sees the whole board at once. It propagates constraints across the full block on each denoising pass and self-corrects before the output is finalized.
I think this is where the long-term significance of diffusion LLMs gets underestimated, even in the launch coverage. The speed numbers get the headlines. The bidirectionality is the more interesting property.
Your codebase is harder than a Sudoku board. Every function is constrained by the types it returns, the APIs it calls, the contracts it implicitly assumes across files. Code infilling (filling in a function body given what comes before and after) is structurally this exact problem. AR models handle infilling through a special fill-in-the-middle training objective, which is a workaround for the directional constraint. DiffusionGemma handles it architecturally. Same problem class, different layer of the fix. The same pattern shows up in SQL schema migrations, config file generation, anything where you're filling structure with hard constraints on both sides. A migration adding a column needs to be consistent with both the existing schema and the downstream queries that reference it. AR generates left to right without reconsidering earlier choices based on later constraints. You can work around this with careful prompting and multi-pass generation. Workarounds, every one of them.
I spent 3 months chasing subtly inconsistent type signatures across files in Claude Code sessions. The context window was splitting the relevant contracts between 2 sessions. Bidirectional attention within a generation block doesn't fully solve cross-file coherence, but it shifts where the inconsistency originates. Different problem, different fix.
When to Switch, When to Stay
The arbitrage for a builder running Claude Code daily looks like this.
DiffusionGemma makes sense for tasks where the bottleneck is throughput on short-to-medium outputs with structural constraints: boilerplate generation in batch, code infilling where the surrounding context is available, filling structured templates (API schemas, config files, migration stubs), rapid iteration cycles where you're regenerating 10-20 variants under 4,000 tokens. The model is out now, open weights under Apache 2.0, vLLM ready, deployable on RTX 4090. Variable API cost on those tasks goes to zero. The economics shift in a specific and real way for high-repetition local workflows.
Stay on Claude for multi-step reasoning chains, debugging that requires tracing logic across many steps, architecture decisions that need large coherent context, any production task where you need more than 8K tokens in a single pass, anything in the reasoning-heavy tier where that 15-20 point delta is the difference between a useful output and a plausible-looking wrong answer.
The context window constraint is the real practical limiter. 8,192 tokens sounds like a lot until a single moderately complex file takes 3,000 of them. That's not a fine-tuning problem. It's baked into the current generation block size. Future versions will push this up. For now it makes DiffusionGemma a task-specific tool, not a general drop-in.
Karen from Accounting would ask whether this justifies buying a second GPU. The honest answer: if you're already running a local model stack on an RTX 4090, it's a pull-and-test situation, not a hardware decision. If you're starting from nothing, the breakeven on dedicated hardware vs API credits requires actual throughput numbers from your real workflow, not enthusiasm about the benchmark π . The JAX fine-tuning recipe in the developer guide is documented enough that a 500-sample SFT experiment on a specific domain is a weekend project (more achievable than "I'm going to rewrite this in Rust this weekend" anyway).
On the infra side: if you're already routing tasks across different model backends, the pattern behind building CLI-native agents for throughput-sensitive workloads gets more relevant with a local diffusion backend in the mix. DiffusionGemma slots cleanly into that architecture.
The Assumption You Never Questioned
I have a WooCommerce integration in my pipeline that parses distributor CSV feeds in a format that hasn't changed since 2019. I've rebuilt the surrounding infrastructure 3 times. The CSV parser is still the same function, same column order, same regex workaround for an edge case I found in 2021. Nobody touches it because it works. The question "should this still be a CSV parser in 2026" has never been asked. At some point it stopped being a decision and became furniture.
Every stack has furniture.
The pattern shows up every time a technical constraint stays stable long enough to become invisible. In 2023, local inference meant loading a 7B model and watching tokens arrive at 3 per second. The latency made it useless for anything interactive. Developers tried it, found it impractical, switched to API calls, and the decision solidified: local inference is for hobbyists, real work goes through the API. What nobody encoded in that decision was the expiration date. "Local inference is slow" sounds like a fact about physics. "Local inference on 2023 hardware with 2023 models was too slow for that use case" is a claim about a specific context, and specific contexts change.
AR wasn't chosen over diffusion because someone ran a comparison and concluded it was better. It was chosen because diffusion text generation wasn't viable. The assumption "we use AR" was a pragmatic constraint that became invisible the moment it stopped being contested.
If you're working through which defaults in your stack are worth revisiting, how I made routing decisions intentional with prompt contracts is where I started. Or if the stack itself is newer territory, Vibe Coding, For Real covers building on explicit principles from the start, available free on Kindle Unlimited.
For the builders: if your workflow has a repetitive generation layer with structural constraints, start with the Sudoku fine-tuning recipe in the developer guide. Run it, look at what changes between 0% and 80% accuracy, and ask what that implies for your own constraint-heavy tasks.
The routing decision is now a real architecture decision: not which API is cheaper this month, but which structural constraint this model can resolve that the other architecturally cannot.
Sources
- DiffusionGemma: The Developer Guide, Google Developers Blog, June 10, 2026
- Run DiffusionGemma on NVIDIA, NVIDIA Technical Blog, June 10, 2026
- DiffusionGemma benchmarks, Unsloth, June 2026
- Google's DiffusionGemma: first open diffusion release from a tier-one lab, Decrypt, June 10, 2026
This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).
