Natural Language Processing (NLP)

Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding

This paper introduces a novel approach to optimize speculative decoding for Large Language Models (LLMs) by trimming the draft model's vocabulary. The core idea is to balance token coverage (how many tokens the draft model can accurately predict) with draft model latency (how fast it generates tokens). By observing that domain-specific workloads use only a fraction of the full vocabulary, the authors formulate vocabulary selection as a constrained optimization problem. They use a Tree-structured Parzen Estimator (TPE) to find the optimal vocabulary size that maximizes a utility function, which combines token coverage (from training data) and latency reduction (estimated using architecture-aware FLOPs). Experiments show that this method significantly improves speculative decoding throughput, with latency reductions of up to 16% and throughput gains of up to 20% on domain-specific tasks, and up to 6.7% gains on out-of-distribution tasks, while reducing vocabulary size by up to 97%. This demonstrates that a carefully trimmed vocabulary can accelerate LLM inference without sacrificing practical coverage.