Enterprise AI Analysis

ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference

On-device running Large Language Models (LLMs) is crucial for preserving user privacy and enabling ubiquitous AI. However, state-of-the-art frameworks frequently fall back to CPU/GPU for attention operations due to quantization sensitivity. This degrades user experience and complicates system scheduling. This paper introduces ShadowNPU, a novel system-algorithm co-designed sparse attention module, minimizing reliance on CPU/GPU by selectively computing attention on only a tiny fraction of important tokens. ShadowNPU efficiently hides the overhead of identifying crucial tokens using an NPU-based pilot compute, and further optimizes performance through NPU compute graph bucketing, a head-wise NPU-CPU/GPU pipeline, and fine-grained per-head sparsity ratios. This approach delivers superior performance with highly constrained CPU/GPU resources, achieving on-par accuracy with significantly reduced resource consumption compared to existing solutions.