Cross-Domain Uncertainty Quantification

Learn Then Test (LTT) for Monotone Risks

Our analysis highlights Learn Then Test (LTT) fixed-sequence testing as a major improvement. By exploiting the monotone decreasing property of risk R(τ) (higher selectivity means fewer cached errors), LTT eliminates the ln K factor from the correction term, leading to significantly tighter bounds. For instance, on MASSIVE at α=0.10, LTT improved guaranteed coverage from 73.8% (Hoeffding) to 94.0%.

Exact Binomial and Betting-Based Bounds

For binary losses, Clopper-Pearson provides an exact upper confidence bound, proving approximately 2x tighter than Hoeffding for low empirical risks. Our work also evaluates WSR Betting, a fundamentally different approach that constructs a martingale wealth process. WSR betting adapts to the observed loss distribution, provably yielding tighter bounds than traditional concentration inequalities for bounded random variables.

Robustness to Distribution Shift & Tail Risks

We investigate Wasserstein DRO for guarantees under distribution shift and CVaR Tail-Risk Bounds for protection against elevated error rates in subpopulations. While more conservative by design, these bounds offer critical assurances for specific deployment scenarios where robustness is paramount.

PAC-Bayes for Data-Scarce Domains

In target domains with small calibration sets (n ≤ 200), Hoeffding-family bounds become too loose. PAC-Bayes bounds offer a tighter alternative when an informative prior, such as a risk profile from a data-rich source domain, is available. By leveraging the 1/n rate, PAC-Bayes can rescue feasibility in small-n settings where other bounds fail.

Introducing Transfer-Informed Betting (TIB)

Our primary theoretical contribution is Transfer-Informed Betting (TIB). TIB combines the adaptive power of betting-based bounds with cross-domain transfer by warm-starting the WSR wealth process using a source domain's risk profile. This overcomes the "cold start" limitation of standard WSR, achieving tighter bounds in data-scarce settings. We formally prove TIB's validity, dominance over standard WSR when domains match, graceful degradation under divergence, and optimality among plug-in priors.

On NyayaBench v2, TIB achieved 18.5% coverage at α=0.10, representing a 5.4x improvement over LTT + Hoeffding and outperforming PAC-Bayes transfer, demonstrating its significant practical utility in scenarios with limited target data.

Selective Prediction vs. Conformal Prediction

A critical distinction for enterprise deployment is between prediction-set guarantees (Conformal Prediction) and single-prediction risk control (Selective Prediction/RCPS). While conformal methods guarantee the true class is in a prediction set, they often yield multiple candidate classes (e.g., avg. 1.67 classes at α=0.10 on MASSIVE). For applications requiring a single, definitive action, such as agentic caching, RCPS's single-prediction risk guarantee is the appropriate framework.

Progressive Trust Model for Agentic Systems

Our guarantees formalize a progressive trust model for AI agents. As calibration data accumulates, the RCPS certificate tightens, allowing systems to graduate from LLM-supervised (low trust) to semi-autonomous and then fully autonomous execution (high trust). LTT, for example, enables semi-autonomous operation (≈62% coverage) at n≈150 examples, and autonomous operation (≥92% coverage) at n≈400, a significant acceleration over traditional methods.