Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference

雷达摘要

Speculative decoding accelerates LLM inference by having a small draft model propose multiple tokens that a larger target model verifies in parallel, reducing decoding iterations while preserving output accuracy. Five guidelines help select the optimal draft length and mechanism across the Pareto frontier: push GEMMs into the compute-bound region without in…