Enterprise AI Analysis

The AGORABENCH Benchmark

AGORABENCH is a new, comprehensive benchmark designed to test LLM negotiation capabilities across diverse, economically motivated market regimes. It features both single- and multi-product settings, explicitly designed to surface complex commercial negotiation challenges where the seller's initial ask always exceeds the buyer's willingness to pay (WTP).

The benchmark encompasses nine distinct market environments:

• Vanilla Market: Baseline negotiation without complicating factors.
• Deceptive Market: Parties may misrepresent information, testing agents' ability to identify reliable signals. This layer underlies most other non-vanilla markets.
• Monopoly Market: Single-seller environment, analyzing bargaining under asymmetric power.
• Installment Possible Market: Introduces deferred or staggered payments, adding time-sensitive financial trade-offs.
• Negative Perception Market: Seller carries reputational disadvantage, biasing buyers towards lower offers.
• Single Product Market: Negotiation over a single item, isolating price-based outcomes.
• Multi Product Market: Buyers consider multiple items and can substitute, highlighting trade-offs between product preference and cost savings. This condition layers atop non-vanilla markets.

Observed LLM Negotiation Behaviors

Our experiments reveal several emergent behaviors of contemporary LLMs (GPT and Gemini series) in bargaining scenarios:

• Anchoring Effect and First-Mover Advantage: Higher initial anchor prices from the seller generally led to higher final deal prices. Conversely, when the buyer initiated the first offer, average deal prices were lower.
• Irrational Concessions: Smaller models (e.g., gpt-4o-mini) exhibited unstable anchoring, backtracking on stated stances with strictly lower counteroffers after proposing a price, which is atypical in human bargaining and signals inconsistent willingness to pay.
• Market Condition Impact:
• Deceptive: Generally improved buyer outcomes and deal rates.
• Monopoly: Consistently harmed buyers, leading to lower deal rates and MERIT scores due to increased seller leverage.
• Installment: Mixed effects; increased deal rates in single-item settings but often at the cost of worse buyer prices. In multi-item settings, added complexity reduced deal rates.
• Negative Perception: Consistently reduced deal rates, particularly acute in single-item negotiations.
• Intra-series Compatibility: Models generally achieved higher deal rates when negotiating with opponents from the same model series, suggesting shared architectural or training paradigms foster compatible negotiation styles.

MERIT-Guided Prompting & Training

To address LLM misalignment with human preferences, we propose MERIT-guided In-Context Learning (ICL-MF) and fine-tuning strategies:

• ICL-MF Performance: ICL-MF consistently outperformed baselines like ReAct and OG-Narrator across both single and multi-product settings, yielding significant gains in MERIT scores and deal rates. It balances efficiency and effectiveness through appropriate negotiation lengths.
• Opponent-Aware Reasoning (OAR): MERIT guidance encourages LLMs to transition from simple tactic-centric thoughts to deep OAR. ICL-MF agents explicitly hypothesize the seller's hidden costs and calculate economic metrics (CS, NP), leading to more robust outcomes. Applying an explicit OAR prompt to baseline ReAct also substantially elevated its performance, confirming OAR's critical role.
• Human Preference Dataset Training (SFT): Fine-tuning gpt-oss-20b using LoRA on human-preferred dialogues from the deceptive regime also significantly improved performance. SFT proved particularly superior in multi-product negotiations, where stable management of diverse products and complex trade-offs is crucial.

These strategies demonstrate that leveraging human negotiation strategies leads to more robust outcomes than pure reasoning-based approaches.