P-EAGLE: Boost LLM Inference with Parallel Speculative Decoding in vLLM
P-EAGLE is an advanced parallel speculative decoding method designed to significantly accelerate Large Language Model (LLM) inference, integrated into vLLM. It addresses the inherent bottleneck of traditional speculative decoding techniques like vanilla EAGLE, which require sequential forward passes for each draft token. P-EAGLE revolutionizes this by generating all K draft tokens in a single forward pass, thereby removing the sequential overhead and enabling deeper speculation without performance degradation.
Key features include a two-step architecture: prefilling, where the target model generates a token and captures hidden states, and the P-EAGLE Drafter, which constructs parallel inputs for subsequent tokens. For future tokens, it utilizes learnable mask token embeddings and shared hidden states. To handle the increased memory demands of training on long sequences (e.g., 10,800 tokens for GPT-OSS 120B), P-EAGLE employs a sequence partition algorithm and gradient accumulation across sequence chunks.
Integrated into vLLM from version 0.16.0, P-EAGLE manages complex batch metadata and hidden state propagation through a highly optimized, fused Triton kernel. This kernel efficiently populates the drafter's input batch on-GPU, inserting mask tokens and generating metadata in a single pass, offsetting overheads. It also meticulously handles KV cache slot mapping and extends CUDA graph capture ranges.
The benefits are substantial: P-EAGLE delivers up to 1.69x speedup over vanilla EAGLE-3 on real workloads using NVIDIA B200 GPUs, with 5-25% sustained gains at high concurrency. It consistently achieves higher throughput, peaking at a speculation depth of K=7, compared to K=3 for vanilla EAGLE-3. Moreover, P-EAGLE boasts a higher acceptance length (AL), meaning more drafted tokens are accepted by the verifier, directly boosting effective output tokens per second. For instance, it shows a 30% higher AL on HumanEval at K=7.
Targeted at LLM developers and organizations deploying production LLMs, P-EAGLE offers a clear path to maximum inference performance and reduced latency. Pre-trained P-EAGLE drafter heads, lightweight 4-layer models, are readily available on HuggingFace for models like GPT-OSS 120B, GPT-OSS 20B, and Qwen3-Coder 30B, allowing users to enable it with a simple `parallel_drafting: true` configuration in vLLM.
P-EAGLE represents a significant advancement in ai automation llm technology by enabling faster text generation through innovative parallel processing techniques.
As organizations increasingly adopt chatgpt automation llm solutions for production workloads, optimizing inference speed becomes critical for maintaining responsive user experiences.
