Artificial Intelligence in Hand Surgery

Quantifying AI's Diagnostic Prowess

Gemini consistently led in performance metrics, achieving 86.4% Union Accuracy with surgeon recommendations, the highest among all AI models. It also secured the highest average surgeon agreement at 45.5% and top F1, precision, and recall scores (62.0%, 64.0%, 60.0% respectively).

ChatGPT followed closely with 81.8% Union Accuracy and strong overall performance. The moderate to high inter-AI agreements (e.g., 78.0% for Gemini-Perplexity) indicate a certain level of convergence in AI decision-making, though significant variability still exists across different models.

LLM Architecture and Training Insights

The performance differences observed among the AI models can be partly attributed to their underlying architectures, medical fine-tuning, and known limitations. Models with more extensive or specific medical training, such as Gemini (limited fine-tuning on medical datasets) and Perplexity (optimized for research-based queries), generally performed better.

In contrast, general-purpose AIs like Copilot, not specifically trained for medicine, showed lower accuracy in medical decision-making. This highlights the importance of domain-specific training and validation for AI tools in healthcare.

LLM Feature & Limitation Comparison

Navigating Subjectivity and Agreement

The inter-rater reliability analysis among expert hand surgeons revealed moderate to low agreement (Fleiss' Kappa score of 0.122), underscoring the inherent subjectivity in Dupuytren's disease management. Treatment choices are influenced by surgeon experience, interpretation of disease severity, and patient-specific factors.

AI systems mirrored this variability, performing better in cases with unanimous surgeon consensus but showing a consistent decline in accuracy in non-unanimous, presumably more complex, scenarios. This highlights that while AI can replicate expert consensus in clear-cut cases, its reliability diminishes when clinical opinions diverge.

Challenges and the Road Ahead for AI in Hand Surgery

Despite promising results, AI models exhibited notable inconsistencies, especially in complex cases requiring nuanced clinical judgment. Some models demonstrated an overly conservative approach, while others proposed premature interventions, both posing clinical risks.

These findings emphasize the critical need for human oversight and rigorous validation before AI systems can be reliably integrated into surgical practice. Future research should focus on enhancing AI contextual awareness, reducing bias, and ensuring alignment with best practice guidelines, moving towards AI as a complementary decision-support tool rather than a replacement.