Technical Report Analysis

Pre-Training Methodologies

The pre-training of LongCat-Flash follows a robust multi-stage curriculum, focusing on scalability, stability, and agentic capability. It incorporates hyperparameter transfer, model growth initialization, and a multi-pronged stability suite to ensure efficient and reliable large-scale training.

Hyperparameter Transfer & Model Growth: LongCat-Flash leverages hyperparameter transfer based on width scaling and model growth initialization (layer stacking) to efficiently train large-scale models. This strategy significantly reduces computational costs and provides improved performance compared to random initialization, as evidenced in Figure 5b. The model starts as a half-scale version, pre-trained on billions of tokens, then expands.

Training Stability Measures: Training stability is enhanced through router stability control (balancing LM and LB losses), activation stability via hidden z-loss, and optimized Adam's epsilon. Hidden z-loss (Eq. 10) prevents massive activations and loss spikes (Figure 6), while Adam's epsilon is set to 1e-16 to maintain adaptive properties for large-scale models (Figure 7).

Long Context Extension & Decontamination: A two-stage context length extension strategy expands the context window from 8k to 128k tokens, using naturally occurring long-text data and curated source code. Rigorous decontamination procedures, including 13-gram overlap and semantic similarity checks, prevent data leakage from common benchmarks, ensuring robust evaluation confidence.

General Pre-Training Data Strategy: A multi-phase pipeline ensures data quality and diversity, with Content Extraction, Quality Filtering, and Deduplication steps. The data mixture progressively increases the proportion of high-quality reasoning data (e.g., STEM and code), aiming for comprehensive foundational capabilities.

Training Infrastructures & Efficiency

The training infrastructure for LongCat-Flash is designed for scalability with precision, ensuring deterministic computation and efficient distributed training. Key innovations include numerical precision control, kernel optimizations, and advanced distributed strategies like ScMoE for computation-communication overlap.

Numerical Precision Control & SDC Detection: LongCat-Flash employs ULP (Unit in the Last Place) evaluation to quantify floating-point errors and integrates an on-chip, in-place operator recomputation mechanism for Silent Data Corruption (SDC) detection. This ensures bitwise-aligned loss values and minimizes numerical errors during BF16 training, crucial for stable large-scale training. Table 4 demonstrates GEMM precision comparison, highlighting efforts to reduce ULP errors.

Kernel Optimizations for Determinism & Performance: Custom kernel redesigns address determinism overhead, including a deterministic FAG (FlashAttention Gradients) kernel (1.6x faster than original deterministic, 0.95x non-deterministic) and a hierarchical reduction algorithm for Deterministic ScatterAdd (performance parity with non-deterministic). Optimized Grouped GEMM and Fused GemmAdd further enhance efficiency.

Distributed Strategy for Large-scale Training: The training architecture is centered on Expert Parallelism Groups (EP), with Context Parallelism (CP) for attention layers and EP partitioning for FFN layers. ScMoE enables dispatch/combine communication to overlap with dense FFN computation by dividing MoE layers into chunks, significantly reducing non-overlapping communication. Figures 8 and 9 illustrate the ScMoE layer chunking and overall overlapping strategy, which reduced non-overlapping communication from 25.3% to 8.4%.

Figure 8: ScMoE Layer with Chunk achieves highest efficiency through computation-communication overlap.

Figure 9: An overview of LongCat-Flash's overlapping strategy, leveraging ScMoE to maximize efficiency.

Inference and Deployment Optimizations

LongCat-Flash's inference system is optimized through model-system co-design, achieving high throughput and low latency. Key techniques include Single Batch Overlap (SBO), speculative decoding with Multi-Token Prediction (MTP), KV cache reduction, multi-step overlapped scheduling, custom kernels, and fine-grained quantization.

Model-Specific Inference Optimization: A Single Batch Overlap (SBO) scheduling strategy optimizes both latency and throughput by orchestrating computation-communication overlap within the ScMoE architecture. Speculative decoding employs Multi-Token Prediction (MTP) as a lightweight draft model (90% acceptance rate), and Multi-head Latent Attention (MLA) significantly reduces KV cache size and bandwidth pressure.

Minimize Schedule Overhead: A multi-step overlapped scheduler launches kernels for multiple forward steps in a single iteration, effectively hiding CPU scheduling and synchronization. This ensures continuous GPU occupancy and dynamically pre-allocates KV cache slots, guaranteeing convergence in allocated KV cache size even without prior knowledge of accept length, as shown in Figure 10.

Figure 10: Multi-step overlapped scheduler for efficient inference.

Theoretical Decoding Performance: LongCat-Flash achieves significant theoretical improvements in both throughput and latency due to its reduced layer count and the SBO overlapping strategy. The theoretical Time-Per-Output-Token (TPOT) is significantly lower than competing models, demonstrating superior efficiency.