π A curated list of Awesome Parallel Decoding papers to accelerate the inference of (autoregressive) LLMs.
Autoregressive (AR) decoding generates one token per forward pass, leaving the strict token-by-token dependency as the dominant latency bottleneck of modern LLM serving. Parallel decoding refers to the broad family of techniques that shorten this critical path by introducing parallel candidates, parallel verification, parallel reasoning steps, or parallel task orchestration, while ideally preserving the quality (and, for lossless methods, the exact output distribution) of the original model. This repository organizes the literature along the granularity of parallelism β from (i) individual tokens, to (ii) reasoning sequences, to (iii) whole tasks β and additionally tracks the (iv) model architecture-level and (v) system-level improvements. We also introduce (vi) hybrid advances and (vii) related benchmarks that make parallel decoding practical.
Legend. Venue is given as the publication venue (e.g.,
ICML'24,NeurIPS'25); preprints usearXiv'YY. The bracketed token at the start of a title is the method's alias/acronym (e.g.,[EAGLE]). Author and affiliation fields are filled where reliably verifiable and markedN/Aotherwise. A blank Code cell means no official public code repository is currently known.
- πΊοΈ Overview: A Taxonomy of Parallel Decoding
- π 1. Token-Level Parallel Decoding
- π 2. Sequence-Level Parallel Decoding
- π 3. Task-Level Parallel Decoding
- π 4. Architecture-Level Parallel Decoding
- π 5. System and Infrastructure
- π 6. Hybrid Methods
- π 7. Benchmarks and Evaluation
Within each section, papers run from older to newer; click a paper to jump to its full row.
Token-level methods shorten autoregressive generation by proposing, organizing, or verifying multiple token candidates or future positions in parallel.
An independent draft model proposes candidate tokens that the target LLM verifies; this includes foundational speculative-decoding frameworks, drafter training and distillation, and cross-architecture draft adaptation.
The target model produces its own drafts through layer skipping, early exit, attached prediction heads, or internal feature prediction, removing the need for a separate full draft model.
Drafts come from retrieved text, n-gram or suffix caches, reusable context patterns, or parameter-free fixed-point/Jacobi procedures.
The main contribution concerns candidate-tree structure, multi-draft coupling and acceptance, adaptive draft length, verification order, or theoretical properties of speculative sampling.
Speculative decoding is integrated with another technique or application domain, including multimodal generation, quality/alignment/safety controls, watermarking, and accelerated reinforcement-learning rollouts.
Sequence-level methods expose parallelism over semantic units, reasoning segments, partial chains, or complete candidate trajectories.
Several reasoning continuations or complete answer sequences are generated in parallel and then verified, corrected, selected, or aggregated at the semantic level.
One output is decomposed into structurally independent sections, subproblems, or graph nodes that can be generated concurrently.
Parallel test-time compute is allocated across branching reasoning paths, with explicit control of exploration, pruning, early termination, and answer aggregation.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2024.02 | More Agents Is All You Need | Junyou Li et al. (@Tencent) | [code] |
TMLR'24 |
| 2024.10 | [Speculative Rejection] Fast Best-of-N Decoding via Speculative Rejection | Hanshi Sun et al. (@CMU & University of Virginia & UC Berkeley & Princeton & Fudan University & Google DeepMind) | [code] |
NeurIPS'24 |
| 2025.02 | [DPTS] Dynamic Parallel Tree Search for Efficient LLM Reasoning | Yifu Ding et al. (@Beihang University & NTU & Wuhan University) | [code] |
ACL'25 |
| 2025.05 | [Group Think] Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularity | Chan-Jan Hsu et al. (@MediaTek Research) | arXiv'25 | |
| 2025.07 | Adaptive Termination for Multi-round Parallel Reasoning: An Universal Semantic Entropy-Guided Framework | Zenan Xu et al. (@Tencent & CUHK) | arXiv'25 | |
| 2025.09 | [A2R] An Asymmetric Two-Stage Reasoning Framework for Parallel Reasoning | Ziqi Wang et al. (@USTC & Baidu & USYD) | arXiv'25 | |
| 2025.09 | [Parallel-R1] Towards Parallel Thinking via Reinforcement Learning | Tong Zheng et al. (@Tencent AI Lab & University of Maryland & UNC Chapel Hill & City HKU & Washington University in St. Louis) | [code] |
ICLR'26 |
| 2025.09 | [ATTS] Asynchronous Test-Time Scaling via Conformal Prediction | Jing Xiong et al. (@HKU & Huawei & UCL & CUHK) | [code] |
ICLR'26 |
| 2025.10 | Parallel Test-Time Scaling for Latent Reasoning Models | Runyang You et al. (@PolyU & USTC & HIT Shenzhen & Shandong Jianzhu University) | [code] |
ACL'26 |
| 2025.10 | [DeepPrune] Parallel Scaling without Inter-trace Redundancy | Shangqing Tu et al. (@THU & ShanghaiTech University) | [code] |
Findings-ACL'26 |
| 2025.11 | [DTS] Enhancing Large Reasoning Models via Decoding Tree Sketching | Zicheng Xu et al. (@Johns Hopkins University & UNC Charlotte & Rice University & University of Minnesota) | [code] |
ICML'26 |
| 2026.02 | [Parallel-Probe] Towards Efficient Parallel Thinking via 2D Probing | Tong Zheng et al. (@University of Maryland & Washington University in St. Louis & UNC Chapel Hill & City HKU & Tongji University) | [code] |
ICML'26 |
Task-level methods coordinate concurrent agents, plans, tool calls, functions, or environment actions.
Multiple LLM agents or teams solve and deliberate concurrently, then combine their results through voting, synthesis, or coordination.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2023.08 | [AutoGen] Enabling Next-Gen LLM Applications via Multi-Agent Conversations | Qingyun Wu et al. (@Microsoft & PSU & UW & Xidian University) | [code] |
COLM'24 |
| 2024.02 | More Agents Is All You Need | Junyou Li et al. (@Tencent) | [code] |
TMLR'24 |
| 2024.06 | [Mixture-of-Agents] Mixture-of-Agents Enhances Large Language Model Capabilities | Junlin Wang et al. (@Duke University & Together AI & University of Chicago & Stanford University) | [code] |
ICLR'25 |
| 2025.07 | [M1-Parallel] Optimizing Sequential Multi-Step Tasks with Parallel LLM Agents | Enhao Zhang et al. (@UW & Microsoft) | ICMLW'25 |
Independent tools, functions, or predicted future actions are planned, dispatched, or speculatively executed in parallel.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2023.12 | [LLMCompiler] An LLM Compiler for Parallel Function Calling | Sehoon Kim et al. (@UC Berkeley & ICSI & LBNL) | [code] |
ICML'24 |
| 2024.05 | [LLM-Tool Compiler] An LLM-Tool Compiler for Fused Parallel Function Calling | Simranjit Singh et al. (@Microsoft & Southern Illinois University) | arXiv'24 | |
| 2024.10 | Asynchronous Tool Usage for Real-Time Agents | Antonio A. Ginart et al. (@Salesforce AI Research) | arXiv'24 | |
| 2024.12 | [AsyncLM] Asynchronous LLM Function Calling | In Gim, Seung-seob Lee, Lin Zhong (@Yale University) | arXiv'24 | |
| 2025.09 | [DSP] Dynamic Speculative Agent Planning | Yilin Guan et al. (@Johns Hopkins University & University of Alberta & University of British Columbia & Avey Research Center & Google DeepMind & UCSB) | [code] |
ICLR'26 |
| 2025.10 | [Speculative Actions] A Lossless Framework for Faster AI Agents | Naimeng Ye et al. (@Columbia University) | [code] |
ICLR'26 |
| 2025.12 | Optimizing Agentic Language Model Inference via Speculative Tool Calls | Daniel Nichols et al. (@UMD & LLNL) | arXiv'25 | |
| 2026.02 | [W&D] Scaling Parallel Tool Calling for Efficient Deep Research Agents | Xiaoqiang Lin et al. (@Salesforce AI Research) | [code] |
ICLRW'26 |
| 2026.03 | [RealtimeTool] Parallel Decoding for Real-Time LLM Function Calling | Xiaoxin Shi et al. (@SJTU & Shanghai Innovation Institute & Fudan University) | [code] |
ICML'26 |
| 2026.03 | [PASTE] Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving | Yifan Sui et al. (@SJTU & Microsoft & Stevens Institute of Technology & Google & HKUST) | arXiv'26 |
Architecture-level methods redesign training or generation so strict left-to-right token causality is no longer the only decoding path.
The work designs, trains, converts, or scales architectures that support non-left-to-right generation, including masked diffusion, block diffusion, and semi-autoregressive models.
Inference algorithms accelerate trained non-autoregressive or diffusion language models through unmasking policies, confidence schedules, trajectory methods, or error correction.
System and infrastructure work turns algorithmic parallelism into reliable latency, throughput, memory, and SLO gains through serving, scheduling, and algorithmβruntime co-design.
Runtime and serving techniques optimize batching, GPU scheduling, pipelines, distributed execution, caching, or SLO-aware adaptation for parallel decoding.
The decoding algorithm is co-designed with deployment constraints or systems techniques such as quantization, KV-cache management, MoE execution, and hardware-aware verification.
These software projects support speculative or parallel decoding but are not paper records in the taxonomy, so they are not included in the classified-paper counts above.
| Name | Type | Documented Support | Code |
|---|---|---|---|
| vLLM | Serving engine | EAGLE / EAGLE-3, MTP, DFlash, draft-model SD, n-gram, suffix decoding | [code] |
| SGLang | Serving engine | EAGLE-2 / EAGLE-3, MTP, DFlash, draft-model SD, n-gram | [code] |
| TensorRT-LLM | Serving engine | Medusa, EAGLE, ReDrafter, n-gram, draft-model SD, lookahead decoding | [code] |
| TGI | Serving engine | Medusa, n-gram speculation | [code] |
| llama.cpp | Serving engine | Draft-model speculative decoding | [code] |
| NVIDIA Triton | Serving engine | Draft-model SD, EAGLE-3 tutorials | [code] |
| Speculators | Library | Unified abstraction & training for SD drafters (vLLM-native) | [code] |
| TorchSpec | Training framework | PyTorch-native large-scale SD drafter training | [code] |
Hybrid methods make the deliberate combination of multiple decoding or generative paradigms their central contribution.
A non-autoregressive or semi-autoregressive language model drafts multiple tokens in parallel for verification by an autoregressive target.
The draft-then-verify principle is transferred to diffusion language models, model cascades, continuous diffusion models, and visual autoregressive generation.
Related hybrid resource. Speculative Cascades β A hybrid approach for smarter, faster LLM inference (Google Research blog, 2025) connects routing, selection, and speculation into one production-oriented framework.
This layer organizes and evaluates the field through related surveys, benchmark suites, and diagnostic studies.
Related surveys.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2024.01 | Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding | Heming Xia et al. (@PolyU & PKU & MSRA & Alibaba) | [code] |
Findings-ACL'24 |
| 2025.08 | A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models | Lingzhe Zhang, Liancheng Fang et al. (@PKU & UIC & THU & XPENG & Alibaba & HKUST(GZ)) | [list] |
arXiv'25 |
| 2025.10 | A Survey on Parallel Reasoning | Ziqi Wang, Boye Niu et al. (@USTC & Baidu & USYD) | [list] |
arXiv'25 |
Benchmark suites.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2024.01 | [Spec-Bench] A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding | Heming Xia et al. (@PolyU & PKU & MSRA & Alibaba) | [code] |
misc'24 |
| 2025.09 | Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling | Shengyin Sun, Yiming Li et al. (@CityU HK & Huawei) | [code] |
ICLR'26 |
Diagnostic studies.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2025.10 | How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices | Han Peng, Peiyu Liu et al. (@RUC & UIBE & THU & CityU HK) | arXiv'25 | |
| 2025.10 | [ParallelBench] Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs | Wonjun Kang, Kevin Galim et al. (@FuriosaAI & UW-Madison & Microsoft & UC Berkeley & SNU & KRAFTON AI) | [code] |
ICLR'26 |
- hemingkx/SpeculativeDecodingPapers β speculative decoding papers
- zhanglingzhe0820/Awesome-Parallel-Text-Generation β parallel text generation (AR-based & non-AR / dLLM)
- Geralt-Targaryen/Awesome-Speculative-Decoding β speculative decoding (with detailed reading notes)
- hemingkx/Awesome-Efficient-Reasoning β efficient reasoning (speculative reasoning & parallel thinking)
- Xiaohao-Liu/Awesome-Multi-Token-Prediction β multi-token prediction
- PPPP-kaqiu/Awesome-Parallel-Reasoning β parallel reasoning
- wang2226/Awesome-LLM-Decoding β general LLM decoding
- VILA-Lab/Awesome-DLMs β diffusion language models
- ML-GSAI/Diffusion-LLM-Papers - diffusion large language models
- MessiX77/Awesome-Efficient-dLLMs β efficient diffusion language models (dLLMs)
Contributions are very welcome! If you would like to add a paper, fix metadata (venue, affiliation, links), or propose a new category, please open a pull request or an issue. When adding a paper, please keep entries in chronological order within each section and follow the existing table format: Date | [Alias] Title (linked to paper) | Authors (@Affiliation) | Code (+ stars badge) | Venue. Where information is uncertain, please mark it as N/A rather than guessing.
π If you find this repository useful, please consider citing our survey paper: π
@misc{survey-parallel-decoding,
title = {Parallel Decoding of Language Models: A Survey},
author = {Yu-Yang Qian and Jia-Chen Liu and Hao-Cong Wu and Lanxiang Hu and Hao Zhang and Peng Zhao},
year = {2026},
note = {\url{https://github.com/ZinYY/Awesome-Parallel-Decoding}}
}Our survey paper is coming soon, stay tuned!