Enterprise AI Analysis

Ensuring User Safety and Preventing Harm

Safety and harm were central to 51.5% (52/101) of articles. Key aspects include:

• Crisis & Suicidality Management: CAI may provide inappropriate advice or respond inadequately to emergencies, lacking contextual understanding of user cues.
• Dependency & Social Isolation: Constant availability can lead to over-reliance on CAI, potentially increasing social isolation and avoiding human contact.
• Harmful Suggestions: CAI can "hallucinate" or provide inaccurate, harmful information (e.g., weight loss tips for eating disorders, medication advice), diverting users from appropriate care.
• Duty to Warn: Uncertainty about how CAI should adhere to protocols regarding threats of harm to others or child/adult abuse.

Mitigation strategies include human supervision, emergency recognition systems, and restricting free-text input.

Addressing Bias and Promoting Justice

Justice, including bias, inequalities, and discrimination, was raised in 40.6% (41/101) of articles. Concerns include:

• Algorithmic Bias: Systematic errors in CAI design and training data can lead to unfairness, privileging certain groups and providing incorrect information or diagnoses to others.
• Health Inequalities: The "digital divide" (differences in digital literacy, language, internet access) can exacerbate existing health inequalities, limiting who benefits from CAI.
• Epistemic Injustice: CAI's biases can devalue users' utterances, leading to feelings of being unheard and potentially eroding self-confidence.
• Cultural Imposition: Western values embedded in CAI can discriminate against other communities, particularly in mental health disorder manifestations and treatments.

Solutions involve culture-specific design, diverse stakeholder involvement, and avoiding harmful stereotypes in CAI embodiment.

Evaluating CAI Effectiveness

Concerns about CAI's effectiveness or efficacy were discussed in 37.6% (38/101) of articles. Key points:

• Lack of Strong Clinical Evidence: Limited rigorous clinical studies confirm the therapeutic effects of CAI. Many commercial apps lack scientific validation.
• Misrepresentation: Providers often overstate CAI's potential, making it difficult for consumers to discern evidence-based tools from less effective commercial offerings.
• Inherent Limitations: CAI, as a computer program, struggles with human elements like genuine empathy, non-verbal cues, transference, and contextual information, which are crucial for therapeutic outcomes.
• "Trackability Assumption": CAI's ability to accurately track users' feelings and behaviors is questioned, especially if users provide inaccurate input.

Recommendations include further clinical research, clear communication of limitations, and integrating feedback for model training.

CAI vs. Human Therapists: A Comparative Ethical View

Understanding the fundamental differences between Conversational AI and human therapists is crucial for responsible deployment.

Understanding the Scoping Review Methodology

This flowchart illustrates the systematic process undertaken to identify, screen, and analyze articles for this scoping review, ensuring comprehensive coverage of ethical challenges.

Focus: The Underexplored Aspects

While many themes were covered, certain areas require more attention to ensure comprehensive ethical frameworks for CAI.

• Underexplored Themes: "Other themes" were discussed in only 9.9% (10/101) of articles, indicating gaps.
• Stakeholder Perspectives: Patient perspectives and experiences are insufficiently represented, especially in empirical studies. Only 9.9% (n=10) of articles used empirical methods.
• Environmental Impact: The environmental footprint of large language models (LLMs) remains largely underexplored despite growing media attention.
• Normative Analysis: A lack of empirical data and normative recommendations signals opportunities for future research and guideline development.