Healthcare Innovation

This scoping review systematically identified studies evaluating AI-driven CDS tools for ARF in adult patients. The search covered PubMed, CINAHL, and Embase from inception to January 2025. Inclusion criteria required studies to feature an AI-driven CDS tool, test a clinical/human factors/health system outcome beyond model performance, and compare against a control group in a clinical setting. Studies were excluded if they lacked a machine learning model, only assessed model performance, or were conducted in simulated settings. Data extraction and quality assessment followed the DECIDE-AI framework, with a focus on usability and implementation.

Out of 5,987 citations, six studies conducted between 2012 and 2024 were included, involving 40–2,536 patients. Four (67%) focused on predicting weaning from mechanical ventilation. Three (50%) demonstrated statistically significant and clinically meaningful outcomes, such as reduced duration of mechanical ventilation or hospital stay. AI tools varied in algorithms (logistic regression, SVM, XGBoost, LightGBM, ANN) and data input methods (manual vs. automated EHR integration). Three studies described a digital infrastructure for real-time integration and interactive interfaces, but overall, studies met a median of only 3.5 out of 17 DECIDE-AI criteria. Crucially, none reported AI-related errors, malfunctions, or algorithmic fairness, and only one described user characteristics and adherence.

Despite AI evaluation efforts spanning over a decade, few tools have achieved comprehensive clinical evaluation. Studies often focused on predictive accuracy under ideal conditions, neglecting user errors, system malfunctions, or missing data. This limited scope fails to address the complexities of real-world clinical practice, where poor implementation, rather than model limitations, might hinder observed benefits. A significant gap exists in usability evaluation, which refers to the system's ability to help users achieve goals safely and efficiently. Furthermore, implementation reporting was inconsistent, with most studies lacking details on how AI tools integrate into complex healthcare systems, user adherence, and strategies for addressing barriers.

The review reveals a limited body of evidence for AI-driven CDS tools in ARF, with most studies lacking robust clinical evaluations. While early applications in ventilator weaning show promise, inconsistent methodological rigor and reporting gaps hinder broader clinical deployment. Future evaluations must prioritize structured assessments of safety, usability, human-AI interaction, and comprehensive implementation strategies. Such an integrated approach is vital for ensuring AI tools genuinely support clinical decision-making, improve patient outcomes, and achieve sustained adoption in complex healthcare environments.