Developers building AI‑driven services constantly wrestle with the trade‑off between model capability and the cost of running inference at scale. While open‑source LLMs such as Gemma 4 have lowered the entry barrier, the expense of generating each token can still dominate budgets. Google’s latest research release—dubbed multi‑token prediction drafters—promises to slash that cost by roughly three times without sacrificing answer quality. This post unpacks the underlying technique, walks through a practical integration path, and highlights the implications for production workloads.

The rising cost of LLM inference

When an application sends a prompt to a large language model, the model typically produces one token at a time, repeatedly invoking the same forward pass. Even with quantization or pruning, the sheer number of matrix multiplications adds up quickly, especially for high‑throughput services like chat assistants, code completion tools, or real‑time analytics. Open‑source models are attractive because they avoid vendor lock‑in, yet developers still need ways to squeeze more throughput out of the same hardware.

Speculative decoding explained

Speculative decoding is a family of techniques that let a model generate several candidate tokens in parallel, then verify them with a smaller, faster “draft” model. If the draft’s predictions align with the final model’s distribution, the tokens are accepted; otherwise, the system falls back to the traditional step‑by‑step approach. This reduces the number of expensive forward passes required for a given output length.

From single‑token to multi‑token prediction

Early speculative methods focused on predicting a single extra token ahead of time. Google’s new drafter models extend this idea by predicting multiple tokens in one go—hence the name “multi‑token prediction drafters.” By leveraging a lightweight draft network trained to mimic the behavior of the full Gemma 4 model, the system can propose a short sequence (e.g., 3‑5 tokens) that the heavyweight model then validates in bulk. When the draft is accurate, the heavyweight model only needs to run once per block instead of once per token, yielding the reported 3× speedup.

Google’s drafter models for Gemma 4

DeepMind’s engineering team released the drafter models alongside the open‑weight Gemma 4 family. The drafters are distributed as separate checkpoints that can be loaded alongside the main model. According to the team, the approach delivers “up to a 3x speedup without any degradation in output quality or reasoning logic.”

Integration steps for developers

1. Install the updated libraries – Google provides a pip package (gemma-drafter) that bundles the draft checkpoint and utility functions.
2. Load both models – Instantiate the full Gemma 4 model as usual, then load the draft model with the same tokenizer.
3. Wrap the generation loop – Replace the standard model.generate call with drafter.generate, which handles block‑wise speculation internally.
4. Tune block size – Experiment with a draft block length of 3‑6 tokens; larger blocks increase potential speed but may raise the rejection rate.
5. Monitor quality metrics – Track perplexity or downstream task accuracy to ensure the speculative path does not introduce regressions.

The wrapper automatically falls back to the full model when the draft’s predictions diverge, so developers do not need to handle edge cases manually.

Practical workflow example

Imagine a SaaS product that offers on‑demand code explanations. The service receives hundreds of requests per second, each requiring a 150‑token response. Using vanilla Gemma 4, the GPU would be occupied for roughly 150 forward passes per request. With the drafter enabled and a block size of 5, the system reduces the number of heavy passes to 30, freeing compute for additional concurrent users. Benchmarks shared by Google show latency dropping from 120 ms to 40 ms per request on an A100, while maintaining identical BLEU scores on a code‑summarization benchmark.

Performance considerations and trade‑offs

• Draft accuracy – The speed gain hinges on the draft model’s ability to predict the next token block correctly. A poorly trained draft can increase rejection rates, negating benefits.
• Memory footprint – Running two models simultaneously doubles VRAM usage. On memory‑constrained devices, you may need to offload the draft to CPU or use a quantized version.
• Batching behavior – Speculative decoding works best with large batch sizes where the overhead of validation is amortized across many inputs.
• Use‑case suitability – Tasks that require strict token‑level control (e.g., token‑level logits for reinforcement learning) may not benefit, as the speculative layer abstracts away individual token decisions.

Key takeaways

• Speculative decoding lets a lightweight draft model propose multiple tokens, cutting the number of expensive forward passes.
• Google’s multi‑token drafters achieve up to a threefold throughput increase on Gemma 4 without measurable quality loss.
• Integration is straightforward: install the gemma-drafter package, load both models, and replace the generation call with the drafter wrapper.
• Developers should monitor draft rejection rates, memory consumption, and downstream quality to fine‑tune block size.
• The technique is especially valuable for high‑traffic services where inference cost dominates operational budgets.

Conclusion

The open‑source AI community has long sought ways to make powerful models more affordable at scale. By extending speculative decoding to multi‑token predictions, Google provides a practical, drop‑in solution that can dramatically accelerate inference for Gemma 4 and potentially other models in the future. For engineers focused on cost‑effective AI deployment, experimenting with the drafter models is a low‑risk step that could unlock significant performance headroom.

---

Source: The AI tool Google says can speed up LLM inference by 3x