AI chatbots typically generate text one token at a time. A token may be a full word, part of one, or a punctuation mark, and each token requires the model to run another expensive prediction cycle. Speculative decoding is designed to speed up that process without asking the main model to make less careful predictions.
The technique is drawing fresh attention as AI companies put more focus on the cost and speed of serving models, rather than just training larger ones. Alibaba Cloud recently highlighted speculative decoding in a September 2026 explanation of real-time large-language-model inference. Its September 22 hardware announcement also pointed to chips designed for both training and inference workloads.
What speculative decoding is
Speculative decoding uses a draft-and-check method for large language models. A smaller, less expensive model predicts several likely next tokens, then the larger target model evaluates those suggestions together instead of generating every token independently from scratch.
The target model still controls the final response. It accepts the parts of the draft that pass its checks and takes over when the draft no longer matches. Depending on the setup, the draft model may be a separate smaller model or a lightweight component tied to the main model.
How it works
- A draft model quickly suggests a short sequence of possible next tokens.
- The larger model checks that sequence in parallel.
- Tokens that match are accepted, and generation resumes at the first rejected prediction.
That can cut down the number of costly sequential steps needed to generate a response. It works best when the draft model regularly predicts the same tokens the larger model would have produced.
Why it matters
For AI providers, inference is the repeated work involved in producing answers after a model has been trained. Faster inference can improve response times and allow the same computing infrastructure to handle more requests. That affects chat assistants, coding tools, search features, translation systems, and AI agents that may make many model calls during a task.
Speculative decoding doesn’t give an AI model a new capability. It’s a systems optimization for how the model runs. As demand for responsive AI services increases, reducing latency and computing costs is becoming as important as improving the models themselves.
