Research Paper Analysis

Contextualizing Deception in LLMs

Traditional benchmarks often fall short in eliciting sustained deceptive behavior under partial information, a critical aspect of real-world multi-agent scenarios. Social deduction games like Werewolf naturally foster these dynamics, making them ideal testbeds.

Previous studies confirm LLMs' convincing deception capabilities, even when optimized for honesty. GPT-4, for instance, has demonstrated strategic concealment of intentions and fabrication of false narratives. Benchmarks like OpenDeception found over 80% deceptive intent, and MASK revealed that larger models conceal lies more effectively.

However, detection capabilities lag significantly. In benchmarks like The Traitors and Hoodwinked, LLMs that could deceive convincingly were easily deceived themselves. This highlights a critical vulnerability: while LLMs are persuasive, their ability to identify dishonesty in peers is often weak, underscoring the need for improved detection mechanisms.

WOLF Game Loop & Architecture

WOLF implements the social deduction game Werewolf as a programmable state graph with fixed roles: 4 Villagers, 2 Werewolves, 1 Seer, and 1 Doctor. This ensures stable conditions for reproducible comparisons.

The game alternates between a Night Phase (Werewolves eliminate, Doctor protects, Seer investigates) and a Day Phase (players debate, accuse, defend, and vote to exile). Actions resolve simultaneously, with only outcomes (eliminated/survived) revealed.

Debates utilize a bidding system for turn-taking, preventing rigid orderings and simulating spontaneous group dynamics. Crucially, every public statement is treated as a unit of analysis, with self-assessments of honesty and peer-rated deceptiveness, which feed into longitudinal suspicion scores.

Robust Deception Measurement Protocol

WOLF measures deception at the granularity of individual statements. Each statement receives a self-assessment of honesty from the speaker and peer perceptions of honesty from all other players, capturing both deception production and detection in real-time.

Suspicion dynamics are modeled using exponential smoothing (α=0.7) to balance immediate reactions with prior history, mirroring human-like reasoning. This provides dynamic suspicion trajectories.

Statements are analyzed through both self- and peer perspectives using private scratchpads for chain-of-thought reasoning. The taxonomy of deception includes: none (truthful), omission (withholding info), distortion (altering facts), misdirection (diverting attention), and fabrication (introducing false info).

Agents are equipped with role-grounded prompts, ensuring deception arises naturally from incentives. Private scratchpads prevent privileged knowledge leakage, and controlled mechanics standardize interactions for stable runs. Evaluation includes full LLM runs and deterministic subsystem ablations for comprehensive analysis.

Empirical Findings from 100 Simulations

Across 100 games, Werewolves won 70% of games. The benchmark generated 32.4 analyzed statements per game, totaling 3240 events. The self-labeled deception base rate across all roles was 69.4%, indicating that most statements were considered strategically deceptive or not strictly truthful by the speakers themselves.

Per-Role Deception Statistics:

Role	Statements	Self-Deceptions	Avg. Suspicion (%)	Avg. Flagged (%)
Villager	150	107	47.9%	49.8%
Werewolf	74	56	53.3%	60.1%
Seer	61	45	55.1%	61.8%
Doctor	39	27	60.8%

Werewolves attracted higher suspicion (53.3%) and were flagged more often (60.1%) than Villagers (48-50%), but honest roles like Seer and Doctor also drew significant suspicion due to hedging or withholding information.

Cross-Perception Matrix:

Observer \ Target	Villager	Werewolf	Seer	Doctor
Villager	—	46.0%	53.7%	51.7%
Werewolf	50.9%	—	52.7%	49.8%
Seer	49.8%	53.3%	—	53.8%
Doctor	56.4%	58.7%	52.2%	—

Suspicion levels were broadly distributed (46–59%), indicating difficulty in concealing deception entirely. Villager-to-Werewolf suspicion was only slightly higher than Villager-to-Villager, highlighting poor discrimination.

Observer Accuracy:

Role	TP	TN	FP	FN	Accuracy	Precision	Recall	F1
Villager	231	90	52	148	0.58	0.82	0.61	0.70
Werewolf	157	77	79	113	0.53	0.67	0.58	0.62
Seer	144	58	36	139	0.53	0.80	0.51	0.63
Doctor	170	45	46	126	0.53	0.79	0.57	0.66
Overall	702	270	213	526	0.52	0.72	0.48	0.58

Overall accuracy was 0.52, with precision at 71-73% but recall at 48-61%. This high precision, low recall pattern suggests models are cautious when flagging deception but often miss subtle lies.

Temporal Trends: Suspicion towards Werewolves increased from 52% to 62% over rounds, while remaining flat for Villagers and Doctor. The fraction of observers flagging deception also rose from 35% to 47%. This indicates that extended interaction improves the detection of liars.

Deception Type Effects:

Type	Count	Avg. Suspicion (%)	Avg. Flagged (%)
none	426	57.1%	51.4%
omission	129	61.9%	57.5%
distortion	76	50.3%	35.5%
misdirection	51	54.6%	45.5%
fabrication	28	51.7%	40.1%

Subtle deception strategies like omission and misdirection were more successful, while overt fabrications were flagged more reliably, showing models are most vulnerable to less direct forms of dishonesty.

Strategic Insights & Future Directions

WOLF demonstrates a clear asymmetry: LLMs are proficient at deception (Werewolves deceived in 31% of turns) but only moderately effective at detection (52% overall accuracy, 72% precision, 48% recall). Repeated interaction significantly improves the ability to discriminate liars from truth-tellers.

Key findings reveal that suspicion towards Werewolves accumulates over time, diverging from the stable suspicion levels for honest roles. This highlights the importance of longitudinal analysis in deception benchmarks. Subtle deception types, such as omission, prove more effective than overt fabrications, indicating areas for targeted model improvement.

Future research will explore larger, more adaptive games and apply these insights to critical domains like moderation, fact-checking, and negotiation, enhancing the robustness and trustworthiness of AI systems.

Considerations & Reproducibility

To ensure maximum reproducibility, WOLF utilizes fixed role distribution, player sets, and debate lengths. While these controls enable consistent comparisons, they may under-sample more complex, longer-horizon, or coalition-based strategies.

A notable limitation is that deception labels rely on model self-assessments rather than human judgments. While peer analysis provides a valuable cross-reference, the inherent subjectivity of AI self-evaluation may influence results.

WOLF provides a stylized Werewolf environment that effectively captures hidden roles and persuasion under partial information. However, the results may not fully generalize to all real-world multi-agent scenarios. To support auditing and further research, all prompts, outputs, and logs are publicly released.