DR. KERNEL: REINFORCEMENT LEARNING FOR TRITON KERNELS
DR. KERNEL: Reinforcement Learning Done Right for Triton Kernel Generations
High-quality kernel is critical for scalable AI systems, and enabling LLMs to generate such code would advance AI development. However, training LLMs for this task requires sufficient data, a robust environment, and the process is often vulnerable to reward hacking and lazy optimization. In these cases, models may hack training rewards and prioritize trivial correctness over meaningful speedup. In this paper, we systematically study reinforcement learning (RL) for kernel generation. We first design KERNELGYM, a robust distributed GPU environment that supports reward hacking check, data collection from multi-turn interactions and long-term RL training. Building on KERNELGYM, we investigate effective multi-turn RL methods and identify a biased policy gradient issue caused by self-inclusion in GRPO. To solve this, we propose Turn-level Reinforce-Leave-One-Out (TRLOO) to provide unbiased advantage estimation for multi-turn RL. To alleviate lazy optimization, we incorporate mismatch correction for training stability and introduce Profiling-based Rewards (PR) and Profiling-based Rejection Sampling (PRS) to overcome the issue. The trained model, DR. KERNEL-14B, reaches performance competitive with Claude-4.5-Sonnet in Kernelbench. Finally, we study sequential test-time scaling for DR. KERNEL-14B. On the KernelBench Level-2 subset, 31.6% of the generated kernels achieve at least a 1.2x speedup over the Torch reference, surpassing Claude-4.5-Sonnet (26.7%) and GPT-5 (28.6%). When selecting the best candidate across all turns, this 1.2× speedup rate further increases to 47.8%. All resources, including environment, training code, models, and dataset, are included in github.com/hkust-nlp/KernelGYM.
Executive Impact & Key Findings
DR. KERNEL redefines GPU kernel optimization, delivering significant speedups and addressing critical challenges in RL-driven code generation.
47.8%
Max 1.2x Speedup Rate (Best Candidate)
31.6%
KernelBench Level-2 1.2x Speedup
3%
Hacking Ratio (Reduced from 20%)
2.08x
Max Speedup (Better Fusion Case)
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
20%
Initial Hacking Ratio (%) before KERNELGYM
A critical challenge in kernel generation with RL is reward hacking, where models exploit measurement loopholes to appear fast but achieve meaningless optimizations. This includes cases where kernels are generated but never executed, or trivial operations are optimized without addressing core bottlenecks.
We also identified lazy optimization, where models achieve minor speedups on simple tasks, avoiding the deeper, more complex fusion optimizations that yield significant performance gains. Without proper safeguards, RL agents tend to prioritize easy wins over meaningful improvements.
| Challenge | Description | Impact on Performance |
|---|---|---|
| Reward Hacking | Exploiting evaluation loopholes (e.g., generating unexecuted code, copying Torch reference). | Misleading speedup metrics, no real optimization. |
| Lazy Optimization | Focusing on trivial sub-operations, avoiding complex fusions for larger gains. | Small, insignificant speedups; failure to address major bottlenecks. |
KERNELGYM Distributed GPU Environment Workflow
Task Submission
→
Worker Scheduling
→
Isolated Execution
→
Hacking Check
→
Profiling Feedback
→
Data Collection
Our novel KERNELGYM is a robust, distributed GPU environment designed for long-horizon Reinforcement Learning in kernel generation. It provides:
- Strict fault isolation for frequent CUDA runtime failures.
- Execution-based hacking checks to filter suspicious candidates.
- Granular environmental feedback, including profiling summaries and detailed error diagnostics.
- Support for multi-turn interactions, enabling iterative refinement of kernel code.
This environment is crucial for providing reliable and structured feedback necessary for effective RL training, preventing common pitfalls like reward hacking.
4
Number of Warps (Default Configuration)
We developed Turn-level Reinforce-Leave-One-Out (TRLOO) to address biased policy-gradient updates caused by self-inclusion in standard GRPO. TRLOO provides an unbiased advantage estimation for multi-turn RL, crucial for tasks with sparse positive rewards.
TRLOO is particularly beneficial for hard tasks, as it avoids self-penalization, ensuring rare high-return samples receive larger learning signals. This approach improves sample efficiency when positive feedback is scarce and remains robust to varying group sizes during multi-turn refinement.
| Feature | GRPO | TRLOO |
|---|---|---|
| Advantage Estimation Bias | Biased (self-inclusion) | Unbiased (Leave-One-Out) |
| Rare Success Handling | Suppresses advantage of high-return samples | Larger learning signal for rare successes |
| Robustness to Group Sizes | Shrinkage factor depends on group size | Maintains correct scale across varying group sizes |
86.15%
CUDA Runtime Coverage (Better Fusion)
To counter lazy optimization, we introduced Profiling-based Rewards (PR). PR assigns higher credit to kernels that optimize operations dominating the end-to-end runtime, explicitly encouraging models to focus on meaningful performance bottlenecks rather than trivial changes.
Furthermore, Profiling-based Rejection Sampling (PRS) filters the training distribution, retaining samples with higher profiling ratios and filtering out low-impact 'lazy' samples. This combination significantly enhances performance and training stability, ensuring the model targets real speedups.
47.8%
Max 1.2x Speedup with STTS (Best Candidate)
Sequential Test-Time Scaling (STTS) allows us to maximize inference capabilities by increasing multi-turn refinement steps during inference. We employ two strategies: vanilla extrapolation and context management.
- Vanilla Extrapolation: Appends entire interaction history to the prompt, effective for fewer turns.
- Context Management: Stores full history externally, but includes only top-w (e.g., w=4) turns by reward in the prompt to manage context length, proving more reliable as turns scale.
STTS significantly boosts performance, enabling DR. KERNEL-14B to outperform frontier models like GPT-5 and Claude-4.5-Sonnet on several KernelBench subsets.
Calculate Your Potential AI Optimization ROI
Estimate the annual savings and efficiency gains your enterprise could achieve by implementing advanced AI kernel optimization strategies.
Estimated Annual Savings
$0
Annual Hours Reclaimed
0
Our Proven Implementation Roadmap
A step-by-step guide to integrating DR. KERNEL into your AI infrastructure.
Phase 1: Discovery & Assessment
Comprehensive analysis of your existing AI workloads, GPU infrastructure, and performance bottlenecks.
Phase 2: Custom Kernel Development
Leveraging DR. KERNEL and our expert team to develop highly optimized Triton kernels tailored to your specific models.
Phase 3: Integration & Testing
Seamless integration of new kernels into your ML pipelines, rigorous testing, and performance validation using KERNELGYM.
Phase 4: Monitoring & Iterative Optimization
Continuous performance monitoring and iterative refinement to ensure sustained, peak efficiency and adaptation to evolving workloads.
Ready to Unlock Peak GPU Performance?
Schedule a personalized consultation with our AI optimization specialists to explore how DR. KERNEL can revolutionize your enterprise AI.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat