CYBERSECURITY ANALYSIS

The CritBench task corpus consists of 81 cybersecurity challenges evaluating agent performance in IEC 61850 Digital Substation environments. These tasks require proficiency in key protocols such as MMS, GOOSE, SV, and IEC 60870-5-104. Tasks range from analyzing static configurations and passive network traffic to active interaction with virtual machines.

Mapping CritBench Tasks to MITRE ATT&CK for ICS

Task Modalities:

Static Configuration Analysis (30 tasks): Agents parse and interpret SCD-files (XML-based) to understand substation topology, enumerate logical nodes, identify protection functions, extract VLAN assignments, and perform security auditing by detecting configuration changes.

Network Traffic Analysis (30 tasks): Evaluate agents' capability to extract intelligence from raw network packet captures. Challenges include auditing clock synchronization in SV, detecting GOOSE configuration anomalies, reconstructing MMS data models, and correlating hardware identities across network layers.

Dynamic System Interaction (21 tasks): Measure agents' ability to autonomously execute actions against a containerized IED environment with real industrial protocol stacks. Tasks involve discovering MMS data models, reading passive SVs, and targeted state alterations such as flipping a logic breaker or modifying protection thresholds.

The flagship model, GPT-5.1, demonstrated the highest proficiency with an overall accuracy of 86.4%. Open-weight models like Qwen3.5 397BA17 (76.6%), GPT-5 mini (75.4%), and MiniMax M2.5 (74.9%) form a distinct capability cluster, reliably handling basic protocol parsing and static file analysis. The smallest model, GPT-5 nano, showed a clear degradation at 43.3%, indicating a capability threshold for navigating IEC 61850 environments.

The specialized tooling provided by CritLayer significantly improves overall model performance. Agents leveraging CritLayer achieved 77.8% success compared to 58.1% without it, demonstrating that domain-specific tools, offering structured outputs and reducing command-guessing, enable more reliable task execution by LLMs.

Performance by Task Complexity:

All models show high proficiency in static analysis tasks, such as SCD parsing and single-step PCAP analysis, achieving near 100% on inventory challenges. However, performance significantly degrades in dynamic tasks requiring multi-step cross-protocol correlation and live IED interaction. LLMs struggle with persistent sequential reasoning, state tracking, and accurate manipulation of live systems without the specialized tool scaffold. Difficulties arise in mapping discovered logical nodes to precise writable functional constraints and interpreting complex industrial protocol structures like IEC 60870-5-104.

Resource Efficiency:

Open-weight and mid-tier models consume substantial token budgets and frequently enter repetitive loops when industrial tools return unfamiliar binary structures. GPT-5.1 is more efficient, using fewer tokens and completing tasks faster (25 seconds median run duration). Financial costs vary; GPT-5 mini offers a competitive mid-tier accuracy at a highly economical $0.005 per run, while GPT-5 nano lacks capability despite its low cost.

Current Limitations:

CritBench currently uses emulated network infrastructure and software-defined protocol stacks, which, while preserving packet structures, do not fully replicate proprietary firmware behaviors. The task corpus primarily covers IEC 61850 terminology and features, not the full spectrum of cybersecurity attacker or defender tasks. Finally, LLM performance is highly sensitive to system prompting and runtime non-determinism, a factor mitigated but not entirely eliminated by multiple independent runs.

Future Work Directions:

• Transition parts of the tasks into Hardware-in-the-Loop (HIL) setups to capture vendor-specific firmware behaviors and physical process interactions.
• Broaden protocol coverage and diversify the vendor/device corpus to represent a wider range of OT environments.
• Investigate advanced agent scaffolds to address more complex, cross-protocol chains that single agents currently find challenging.
• Incorporate prompt-robustness stress tests and standardized evaluation of safety-critical failure modes to better quantify risks as LLM capabilities evolve.