Enterprise AI Analysis
POCKET FOUNDATION MODELS DISTILLING TFMS INTO CPU-READY GRADIENT-BOOSTED TREES
A fraud scorer needs to answer in under 2 ms. The best tabular foundation models (TFMs) take 151 to 1,275 ms on GPU. We close this gap by distilling the TFM offline into an XGBoost or CatBoost student that runs natively on CPU. The central obstacle is specific to in-context learning (ICL) teachers: they leak labels when scoring their own training set, so the soft targets collapse to near-one-hot vectors with no inter-class structure left to distill. Stratified out-of-fold (OOF) teacher labeling prevents this. Across 153 classification datasets drawn from TALENT, OpenML-CC18, TabZilla, and TabArena, distilling TabICLv2 into XGBoost gives 0.882 macro-mean AUC (96.5% of teacher AUC) at 1.9 ms on CPU, a 38× to 860× speedup across teacher-student pairs with a statistically significant edge over a tuned CatBoost baseline (Wilcoxon p=0.0008; 51% win rate). Four further findings: teacher rank transfers exactly to student rank; gains concentrate on low-dimensional data (<21 features: +0.011 over CatBoost vs. >21 features: +0.001); multi-teacher averaging helps MLP students (+0.006, p=0.003) but adds less than 0.001 for tree students; and on high-dimensional tasks where the teacher itself trails CatBoost, distillation makes things worse rather than better. The full pipeline is open-sourced as part of the TabTune library.
Executive Impact & Key Metrics
This research outlines critical advancements in deploying high-performance tabular AI models, highlighting a distillation pipeline that transforms slow, powerful foundation models into CPU-ready, low-latency students. The key contributions are:
- OOF labeling is mandatory for ICL-based TFMs to preserve soft-label signal, preventing collapse to near-one-hot vectors.
- Distilling TabICLv2 into XGBoost achieves 38x-860x latency reduction, beating CatBoost baselines on 51% of datasets.
- Teacher selection is simplified: rank by solo-AUC on a held-out sample, as teacher rank directly transfers to student performance.
- Gains are concentrated on low-dimensional data (fewer than 21 features), with minimal impact on high-dimensional tasks.
- Multi-teacher averaging offers minor benefits for MLP students but is negligible for tree students.
0x
Min. Inference Speedup
0 AUC
Avg. AUC Gain (vs. CatBoost)
0%
Datasets Beating Baseline
0 ms
Student CPU Latency
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Introduction
Tabular Machine Learning faces significant deployment challenges, particularly regarding inference latency for powerful foundation models. While models like TabICLv2, TabPFNv2.6, and LimiX achieve high accuracy, their reliance on large transformers and GPUs leads to latency of 151-1,275 ms per batch. This makes them unsuitable for time-sensitive applications such as fraud alerts, credit scoring, or patient triage. Knowledge distillation offers a solution, enabling the transfer of a teacher TFM's accuracy to a smaller, CPU-ready student model. This paper focuses on addressing a critical obstacle specific to in-context learning (ICL) teachers: the collapse of soft targets to near-one-hot vectors due to label leakage during training, which destroys inter-class structure essential for distillation. The proposed stratified out-of-fold (OOF) teacher labeling technique resolves this issue, making ICL distillation viable and effective.
Related Work
The research builds upon existing work in Tabular Foundation Models (TFMs) like TabPFN, TabPFNv2, TabICLv2, LimiX, TabDPT, and Orion-Bix, which leverage transformers for in-context learning to achieve state-of-the-art performance on tabular datasets. These models often participate in shared benchmark pools such as OpenML-CC18, TabZilla, TALENT, and TabArena. In knowledge distillation, the concept of transferring "dark knowledge" via soft labels was introduced by Hinton et al., with Born-again networks demonstrating that students can match or exceed teachers. While distillation into tree-based students is less common, out-of-fold label collection is standard in tabular stacking. This work specifically adapts OOF labeling to ICL-based teachers to mitigate the known leakage problem, differentiating it from prior tabular model compression efforts like TabNet and GBDT-to-linear transfer that do not target ICL-based TFMs at scale.
Technical Approach
The core of the methodology addresses the issue of label leakage in In-Context Learning (ICL) teachers. When an ICL teacher scores examples present in its own context, its predictions collapse to near-one-hot vectors, eliminating the valuable inter-class structure that soft labels provide for distillation. The proposed solution is stratified out-of-fold (OOF) teacher labeling: the training set is partitioned into K=5 stratified folds, and for each fold, the teacher is fitted on the remaining data and predicts only on that specific fold. This ensures that the teacher never scores an example it has seen during its own training, preventing label leakage. The student model (XGBoost, CatBoost, LightGBM, or MLP) is then trained using a Hinton mixed loss, combining temperature-scaled Kullback-Leibler (KL) divergence on soft targets with cross-entropy on hard labels. Adaptive per-sample temperatures and confidence weights are used to enhance the distillation process. This pipeline ensures reliable and effective knowledge transfer from powerful, but slow, TFMs to fast, CPU-ready student models.
Empirical Results
Experiments were conducted across 153 classification datasets from TALENT, OpenML-CC18, TabZilla, and TabArena, using 4 TFM teachers and 4 student families. The key finding is that distilling TabICLv2 into XGBoost achieves a 0.882 macro-mean AUC, retaining 96.5% of the teacher's AUC, at an inference latency of 1.9 ms on CPU. This represents a 38x to 860x speedup over teacher-student pairs and significantly outperforms a tuned CatBoost baseline on 51% of datasets. Further observations include: teacher AUC rank transfers directly to student rank, simplifying teacher selection; distillation gains are more pronounced on low-dimensional data (fewer than 21 features, +0.011 AUC gain) compared to high-dimensional data (more than 21 features, +0.001 AUC gain); multi-teacher averaging benefits MLP students slightly (+0.006 AUC) but is negligible for tree-based students; and distillation can worsen performance when the teacher itself underperforms a well-tuned GBDT on high-dimensional tasks.
Analysis & Implications
The consistent transfer of teacher rank to student rank signifies that a stronger teacher reliably produces a better student, simplifying the teacher selection process to merely picking the highest solo AUC. Tree students, with their capacity ceiling, effectively absorb the teacher's decision boundary, achieving 97-98% retention of teacher AUC. MLP students, while benefiting from multi-teacher averaging as a form of label smoothing (gaining +0.006 AUC), generally exhibit lower retention (92-94%) and noisier absolute rankings, indicating they under-fit single teachers more. Calibration analysis shows that tree students inherit teacher calibration well (within ~0.01 ECE), whereas MLP students are 0.02-0.04 worse calibrated, a gap mostly recovered with post-hoc temperature scaling. Crucially, the pipeline's utility is predictable, and it fails gracefully when the teacher itself is weak. The mandatory nature of OOF labeling for ICL teachers ensures that the soft labels provide meaningful inter-class structure, allowing students to learn effectively and operate at 1% of the inference cost of the original TFMs.
Enterprise Process Flow: OOF Distillation Pipeline
Partition D into K stratified folds
→
Fit teacher(s) on D\Dk
→
Predict on Dk (OOF labels)
→
Multi-teacher Averaging (if M>1)
→
Set per-sample Temp. and Weight
→
Train student fs
→
Return fs for CPU deployment
38x
Minimum Inference Speedup (Teacher vs. Student)
| Feature | GBDT Baselines (e.g., CatBoost) | Distilled TFMs (e.g., TabICLv2→XGB) |
|---|---|---|
| Accuracy |
|
|
| Latency |
|
|
| Data Type Affinity |
|
|
| Training Data Requirement |
|
|
Real-world Impact: Fraud Detection
Problem: Traditional Tabular Foundation Models (TFMs) like TabICLv2 are powerful but suffer from high inference latency (151-1,275 ms on GPU). This makes them impractical for latency-critical applications such as real-time fraud scoring, where responses are needed in under 2 ms.
Solution: By implementing a stratified out-of-fold knowledge distillation pipeline, these high-performing TFMs can be distilled into lightweight, CPU-ready Gradient-Boosted Trees (e.g., XGBoost). This process captures the "dark knowledge" from the teacher while reducing inference time to just 1.9 ms on CPU.
Benefit: Enterprises can now leverage the advanced accuracy of tabular foundation models for critical, real-time decisions without compromising on speed. This enables proactive fraud detection and other low-latency applications, delivering significant operational and financial benefits.
Calculate Your Potential AI ROI
Estimate the significant time and cost savings your enterprise could achieve by integrating optimized AI solutions.
Estimated Annual Savings
$0
Annual Hours Reclaimed
0
Your AI Implementation Roadmap
A structured approach to integrating Pocket Foundation Models into your enterprise for maximum impact and efficiency.
Phase 1: Teacher Model Selection & OOF Labeling
Identify the strongest TFM teacher (e.g., TabICLv2) suitable for your data. Generate stratified out-of-fold (OOF) soft labels across your datasets on GPU. This crucial step prevents label leakage and ensures the soft targets retain valuable inter-class structure, which is essential for effective distillation. This phase sets the foundation for high-quality student training.
Phase 2: Student Model Training
Train CPU-ready student models, such as Gradient-Boosted Trees (XGBoost, CatBoost) or Multi-Layer Perceptrons (MLP), using the generated soft labels. Employ a Hinton mixed loss, which combines temperature-scaled KL divergence on soft targets with cross-entropy on hard labels. Adaptive per-sample temperatures and confidence weights further optimize the distillation process, allowing students to efficiently learn from the teacher's nuanced predictions.
Phase 3: Performance Benchmarking & Optimization
Rigorously evaluate the trained student models for AUC, inference latency, and calibration across diverse datasets. Analyze performance differences between various teacher-student pairs, paying special attention to gains on low-dimensional data and the impact of multi-teacher averaging. This phase involves fine-tuning student hyperparameters to maximize performance while ensuring real-time inference requirements are met.
Phase 4: Deployment & Integration
Deploy the optimized, distilled student models (e.g., TabICLv2→XGB) natively on CPU for ultra-low latency inference, achieving sub-2ms response times. Seamlessly integrate these "pocket" foundation models into your existing enterprise systems, such as real-time fraud detection, credit scoring, or patient triage. This final step transforms academic breakthroughs into tangible business value, enabling critical, fast decision-making without the overhead of larger GPU-dependent models.
Ready to Transform Your Enterprise with AI?
Don't let latency hold back your advanced AI deployments. Discover how distilling powerful foundation models into efficient, CPU-ready solutions can unlock real-time performance and deliver a competitive edge. Our experts are ready to discuss your specific needs and chart a path to smarter, faster operations.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat