Enterprise AI Analysis

A Catalogue of Evaluation Metrics for LLM-Based Multi-Agent Frameworks in Software Engineering

LLM-based Multi-Agent (LMA) frameworks have recently gained traction in software engineering, attracting attention to their potential to enhance productivity by automating tasks such as code generation, testing, and quality assurance. However, evaluation practices in this area remain fragmented due to the lack of standardised methodologies. Frameworks often rely on self-defined or inconsistent metrics, hindering reproducibility and making frameworks appear optimal within their own settings, which obscures the true state of the art and can produce artificially inflated performance results. To address these challenges, this study conducts a comprehensive analysis of evaluation metrics used by state-of-the-art LMA frameworks, revealing inconsistencies and conceptual gaps. We propose 12 novel metrics, combine them with 26 existing ones, resulting in a structured catalogue of 38 metrics across four technical categories: Outcome, Process, Product, and Framework. These contributions provide a structured foundation for rigorous, reproducible LMA framework evaluation, enabling direct and meaningful comparisons between frameworks and supporting the systematic advancement of LMA frameworks for software engineering.