Enterprise AI Analysis

Identifying Optimal Architectures

A systematic empirical study was conducted, evaluating over 50 configurations across six major ASR model families: OpenAI Whisper (Encoder-Decoder), NVIDIA Nemotron Speech Streaming (Cache-aware Transducer), Parakeet TDT (TDT Transducer), Canary (AED + AlignAtt), Conformer Transducer XL, and Qwen3-ASR (LLM-based ASR). These were tested in batch, chunked, and streaming inference modes.

The evaluation revealed that batch-oriented models like Qwen3-ASR-1.7B and Whisper, while highly accurate offline, suffered significant degradation when adapted to streaming. For instance, Parakeet TDT-0.6B-v3's WER increased by 46% relatively in chunked mode. In contrast, NVIDIA's Nemotron Speech Streaming (0.6B) emerged as the strongest candidate. It is purpose-built for real-time streaming with a cache-aware conformer transducer architecture, enabling flexible latency-accuracy trade-offs without retraining. The Nemotron-0.6B configuration (7, 10, 7) provided the optimal balance, achieving only 0.21% absolute WER degradation from its batch baseline with 0.56s algorithmic delay, confirming its suitability for natively streaming scenarios.

ONNX Runtime & Quantization for Edge ASR

To enable efficient CPU-only inference, the chosen Nemotron-0.6B model was re-implemented within ONNX Runtime. The optimization strategy involved several key design decisions:

1. Three-Graph Decomposition: The model was split into independently optimizable encoder, decoder, and joiner ONNX sessions, allowing per-component quantization and graph-level optimizations like multi-head attention fusion.
2. Stateful Streaming with Zero-Copy Cache Management: An inference loop was designed to update rolling cache tensors and LSTM states in-place between chunks, minimizing memory allocations and copies.
3. Native Mel Spectrogram Extraction: Audio preprocessing, including log-mel feature extraction and ring-buffer pre-encode cache management, was implemented directly in ONNX Runtime for acoustic continuity across chunks.
4. RNNT Greedy Decoding: The inference loop utilized RNNT greedy decoding as a state machine, avoiding the overhead of beam search while maintaining accuracy for streaming.

For further size reduction and performance enhancement, calibration-free weight-only block quantization was applied. The study evaluated Round-To-Nearest (RTN) and a custom importance-weighted k-quant method. K-quant optimizes scale and offset to minimize reconstruction error, focusing on large-magnitude weights. Mixed-precision schemes were also explored, where accuracy-sensitive layers retained higher precision (e.g., int8) while most layers were reduced to int4. The encoder, comprising over 95% of model parameters, was the primary target for quantization, with the decoder and joiner remaining in FP32 to preserve stability.

Breakthroughs in Efficiency & Accuracy

The rigorous optimization pipeline delivered significant improvements, establishing a new Pareto point for on-device streaming ASR:

• Model Size: The Nemotron-0.6B model was reduced from 2.47 GB (FP32 ONNX) to a compact 0.67 GB with int4 k-quantization, representing a 73% reduction.
• Accuracy: Despite aggressive compression, the int4 k-quant variant achieved an 8.20% average WER, showing only a 0.17% absolute degradation (2.1% relative) from the 8.03% ONNX FP32 baseline. The int8 k-quant variant essentially matched FP32 accuracy (8.01%).
• CPU Inference Speed: All ONNX variants achieved a Real-Time Factor (RTFx) greater than 6x on CPU. The int4 k-quant variant specifically achieved 7.20x RTFx, indicating that reduced precision can even accelerate throughput on CPU.
• Low Latency: With the selected (7, 10, 7) streaming configuration, the system achieves a comfortable 0.56s algorithmic delay, leading to an effective time-to-first-token well under 0.7s, dominated by audio accumulation.

These results demonstrate that aggressive 4-bit compression is viable for high-quality streaming ASR on resource-constrained, CPU-only edge hardware, enabling highly efficient and private speech recognition applications.