Enterprise AI Analysis

Our 3-Stage Concept-Level Explanation Generation Approach

The framework operates in three main stages:

1. Decision Path Extraction: Local, feature-based decision paths are extracted from black-box ML models, representing sequential conditions leading to a prediction.
2. Feature-Concept Mapping Generation: Low-level features within these paths are translated into high-level clinical concepts using LLMs enhanced with domain-specific knowledge (nephrology guidelines).
3. Concept-Based Annotation of Decision Paths: The extracted decision paths are annotated with these generated clinical concepts, incorporating "positive transformation" for clarity and "partial coverage scores" to quantify concept satisfaction.

This systematic approach ensures that explanations are complete, correct, and clinically actionable, reflecting evidence-based standards.

LLM-Enhanced Feature-to-Concept Mapping

A key innovation of this work is the use of Large Language Models (LLMs) for generating high-level concepts. Standard LLMs were customized with pre-training on clinical guidelines (e.g., KDIGO, OPTN) and research publications, creating specialized AI assistants like the "Kidney Disease Professor."

These enhanced LLMs analyze granular SRTR dataset features (like diabetes mellitus, hypertension, obesity) and synthesize them into meaningful clinical concepts (e.g., Metabolic Syndrome, Recipient Frailty). A multi-stage validation framework mitigates hallucination risks, ensuring consensus-driven mappings. Concepts are mapped using either strict propositional-logic rules or flexible threshold-based rules (e.g., Recipient Frailty based on 'at least k out of n' conditions). This dual approach balances expressiveness and simplicity, aligning concepts with their clinical definitions.

Illustrative Case Examples

The paper demonstrates the concept-based annotations using Sankey plots, visualizing how low-level features contribute to various high-level and intermediary concepts. Two exemplary decision paths are presented: one leading to all-cause graft loss (Figure 3) and another to patient/graft survival (Figure 4).

For a graft loss scenario, features like prolonged dialysis vintage, specific ESKD diagnosis, low serum albumin, and prolonged cold ischemia time were mapped to concepts such as 'high kidney disease burden' and 'technical complexity high'. The partial coverage scores highlight nuanced contributions, where some concepts are fully met while others show only 'moderate partial' satisfaction due to competing signals. This reveals a more comprehensive and clinically relevant understanding than simple feature importance.

Clinical Significance & Future Directions

This concept-level XAI framework marks a significant step towards integrating advanced AI systems into real-world clinical practice. By providing explanations in a language familiar to clinicians, it addresses the interpretability barrier that often hinders AI adoption. The ability to identify novel scenarios where anticipated risk may not lead to poor outcomes, due to counterbalancing factors, is a crucial clinical implication.

Future work will focus on deeper validation through controlled experiments and prospective studies to measure the impact of concept mapping and partial coverage on clinical decision-making. Refining concept definitions beyond propositional logic and developing standardized weighting mechanisms are also identified as key areas for improvement.