ENTERPRISE AI ANALYSIS

Background

This section outlines key perspectives and developments in reasoning and its evaluation. It reviews major theoretical accounts of human and machine reasoning, surveys existing reasoning benchmarks used to assess AI models' reasoning capabilities across domains, examines approaches to autograding open-ended reasoning performance, and concludes with a discussion of current benchmark limitations and gaps.

Philosophical approaches to reasoning typically present idealized accounts of how reasoning should operate to draw conclusions. A standard account is the classification of reasoning modes into deduction, induction and abduction. Deduction and induction were already recognized and distinguished in ancient Greek philosophy, particularly in Aristotle's logic [6]. Much later in the 19th century Charles Sanders Peirce [7] added abduction to this framework. The modes can defined as follows:

• Deduction: Reasoning following established canonical logical patterns.
• Induction: Inferences based on generalization from previous, ideally repeated, observations.
• Abduction: The best possible explanation for the given facts.

Experimental Methods

The Watson & Holmes (W&H) tabletop game [5] was adapted for use in assessing the reasoning abilities of LLMs. We first describe the original form of the game, and then its adaptation.

Watson & Holmes is a whodunnit game, in which players compete to be the first to solve a mystery using reasoning skills such as deduction, induction, and abduction. Each play of the game involves a fresh case, which can only be attempted once only.

The eleven cases used for performance evaluation had a mean of 15 (SD = 2) locations. The mean word count per case (introduction plus location texts) was 2,900 (SD = 600). The number of questions per case ranged from two to five (mean = 3.4), giving in total 37 questions across the eleven cases.

Results

Results show a clear improvement in AI model performance over time. Over nine months of 2025, model performance rose from the lower quartile of the human comparison group to approximately the top 5%. Around half of this improvement reflects steady advancement across successive model releases, while the remainder corresponds to a marked step change associated with reasoning-oriented model architectures.

Systematic differences in the performance of AI models compared to humans, dependent on features of the specific detection puzzle, were mostly absent with the exception of a fall in performance for models when solving longer cases (case lengths being in the range of 1900-4000 words), and an advantage at inductive reasoning for reasoning models at early stages of case solving when evidence was scant.

Discussion & Conclusion

This study developed and populated a novel reasoning benchmark (WnH) based on the Watson & Holmes tabletop detective game. Several key findings emerge.

First, AI expected performance at the benchmark has improved from lower quartile of our comparison human population (Computer Science undergraduates at a leading university) to top 14% level (95% confidence) over 9 months of 2025. Roughly half of that improvement is incremental with model release date, and half can be attributed to a step change when rModels were introduced.

Finally, the utility of the benchmark is near its end for the frontier LLMs, with saturated performance expected to be achieved by the end of 2026. The benchmark is expected still to remain useful for assessing small, cost-effective AI models.