Enterprise AI Analysis

Evaluating Metrics for Safety with LLM-as-Judges

LLMs are increasingly used in text processing, even for safety-critical tasks like triaging post-operative care. However, LLMs make mistakes, and ensuring their safety and reliability in critical information flows is paramount. This paper argues against solely relying on performative claims about augmented generation frameworks. Instead, it advocates for a safety argument focused on evaluating evidence from LLM processes, particularly those using LLM-as-Judges (LaJ) evaluators. While deterministic evaluations are challenging for many NLP tasks, adopting a basket of weighted, context-sensitive metrics, defining error severity, and designing confidence thresholds for human review can lower error risk. The paper explores an agentic framework for peri-operative risk assessment as a use case, demonstrating how LaJ evaluations can provide assurance evidence by comparing an 'anaesthetist agent's' risk brief against 'clinical specialist agents'' briefs, with an 'adjudicator agent' evaluating concordance based on coverage, critical items, correctness, prioritisation, and actionability, flagging low concordance for human review.