Enterprise AI Analysis
Estimating the odds ratio from the output scores of machine learning models: possibilities and limitations
Ronit Nirel, Naor Bauman, Efrat Morin, Nimrod Maimon & Raanan Raz
Abstract: Estimation of exposure-response association is central to epidemiologic research. Although the advantages of machine learning (ML) techniques for modeling complex relationships are well-recognized, their use in epidemiologic studies are limited mainly because they do not provide direct estimates of associations, such as odds ratios (ORs). We suggest eight hybrid estimators of the OR that are functions of the output from a classifier, or their probability-calibrated form, multiplied by an adjustment factor that is 'borrowed' from logistic regression (LR). We also suggest two estimators based on partial dependence functions. We applied these estimators to output from LR, random forest (RF) and gradient boosting (GB) models for investigating associations between (1) temperature and respiratory or cardiovascular admissions and (2) prenatal exposure to temperature and overweight among infants. Most (87%) of the estimates produced by GB were within the LR 95% CI, but for RF the results were mixed: 0%, 60% and 13% of the estimates were within this CI for the Respiratory, Cardiovascular and Infants data, respectively. Additionally, GB-based CIs for the uncalibrated estimates were narrower by 13-59% compared to the LR CIs. These findings may enhance the integration between ML and epidemiologic research by providing interpretable results.
Executive Impact & Key Metrics
Leverage advanced machine learning for more precise and interpretable epidemiological insights. Our analysis demonstrates a path to integrating complex ML models with traditional statistical measures.
0
GB Estimates within LR 95% CI
0
Reduction in CI Width for GB
0
Hybrid OR Estimators Introduced
0
Avg. ECE Reduction via IsoReg
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Hybrid OR Estimators
This research introduces ten hybrid estimators for the Odds Ratio (OR), designed to integrate the predictive power of Machine Learning (ML) models with interpretable association measures. These estimators are derived from classifier outputs, either raw or probability-calibrated, and incorporate an adjustment factor 'borrowed' from logistic regression. This innovative approach addresses a significant limitation of ML in epidemiological studies by providing direct, interpretable OR estimates.
Two main categories of estimators are proposed: eight hybrid estimators based on various calibration methods (Platt, IsoReg, GUESS) and statistical approaches (GMO, OMP), and two estimators based on Partial Dependence Functions (ParDepFuns). This framework allows for robust estimation even with complex non-linear relationships and interactions inherent in real-world data.
Calibration & Model Consistency
The study rigorously evaluated the performance of these estimators using Logistic Regression (LR), Random Forest (RF), and Gradient Boosting (GB) models across two distinct epidemiological datasets. A critical aspect of the methodology involves probability calibration of ML model output scores. Calibration metrics such as Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and Brier Score (BS) were used to assess the accuracy of predicted probabilities.
Significantly, IsoReg calibration reduced ECE values by an average of 92% for RF and GB models, demonstrating its effectiveness in aligning predicted scores with true probabilities. However, high calibration performance did not always guarantee reliable OR estimates, especially for RF models with mixed results. GB models consistently showed better calibration and more reliable OR estimates.
Interpretable Results for Epidemiology
The core finding is the strong consistency of Gradient Boosting (GB) based OR estimates with those from traditional Logistic Regression (LR). Specifically, 87% of GB estimates fell within the LR 95% Confidence Intervals (CIs). Furthermore, GB-based CIs were often narrower by 13-59% for uncalibrated estimates, suggesting improved precision.
In contrast, Random Forest (RF) models yielded mixed and highly variable results for OR estimation, with consistency ranging from 0% to 60% depending on the dataset. This highlights the importance of careful model selection and calibration when using ML for OR estimation. The ability to obtain interpretable ORs from ML models opens new avenues for integrating advanced AI techniques into epidemiologic research, providing both predictive power and clear measures of association.
87%
of Gradient Boosting (GB) OR estimates were within the Logistic Regression (LR) 95% Confidence Interval.
Enterprise Process Flow: OR Estimation with Partial Dependence Functions
Fit classifier 'g' to original data
→
Create new datasets (X=1, X=0)
→
Obtain scores from classifier
→
Average logit(odds) to get h(1) and h(0)
→
Estimate OR by exp[h(1)-h(0)]
| Feature | Logistic Regression (LR) | Gradient Boosting (GB) | Random Forest (RF) |
|---|---|---|---|
| Consistency with LR ORs |
|
|
|
| CI Width (vs. LR) |
|
|
|
| Calibration Impact |
|
|
|
Application: Admissions Data Case Study
Context: This study examined the association between ambient temperature and the risk of hospitalization for respiratory or cardiovascular diseases among elderly patients in southern Israel (2004-2009). The dataset involved 17,011 admissions.
Challenge: While Machine Learning (ML) models offer high predictive performance for complex relationships, they traditionally lack direct, interpretable measures of association like Odds Ratios (ORs) that are critical in epidemiology.
Solution: We applied hybrid OR estimators, developed in this research, to Logistic Regression (LR), Random Forest (RF), and Gradient Boosting (GB) models. This involved using both uncalibrated and calibrated output scores, alongside partial dependence functions, to derive OR estimates.
Outcome: For the Admissions data, GB-based OR estimates showed strong consistency with LR estimates, with their 95% CIs often being narrower. However, RF models produced mixed and highly variable results, emphasizing that careful selection and calibration of ML models are essential to obtain reliable and interpretable epidemiological findings.
Calculate Your Potential AI Impact
Estimate the efficiency gains and cost savings your enterprise could achieve by implementing AI-driven insights from cutting-edge research.
Estimated Annual Savings
$0
Annual Hours Reclaimed
0
Your AI Implementation Roadmap
A clear path to integrating advanced AI insights into your enterprise, maximizing both interpretability and impact.
Phase 1: Discovery & Strategy Alignment
We begin by thoroughly analyzing your current epidemiological research workflows and identifying specific areas where ML-derived OR estimates can provide significant value and interpretability.
Phase 2: Data Integration & Model Prototyping
Securely integrate your proprietary datasets. Develop and prototype hybrid OR estimators using various ML models (GB, RF) and calibration techniques, ensuring robust performance and consistency with traditional methods.
Phase 3: Validation & Customization
Validate the performance of the selected hybrid OR estimators against your benchmarks. Customize models and calibration methods to optimize for accuracy, interpretability, and specific business needs.
Phase 4: Deployment & Continuous Optimization
Deploy the validated AI solution within your existing infrastructure. Establish monitoring for continuous performance optimization and user feedback, ensuring long-term value and adaptability.
Ready to Transform Your Research with Interpretable AI?
Schedule a personalized consultation to explore how our advanced AI solutions can elevate your epidemiological studies and provide clear, actionable insights.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat