HUMAN-AI ALIGNMENT IN COLLECTIVE REASONING

Simulating Collective Reasoning with "Lost at Sea"

We developed an empirical framework using the well-established "Lost at Sea" social psychology task, known for revealing gender biases in leader selection. Our methodology involved a large-scale online experiment with 748 human participants, grouped into four-person cohorts.

Groups were randomly assigned to two conditions: Identified (HI), where demographic attributes (name, gender, avatar) were visible, or Pseudonymous (HP), using gender-neutral aliases. Participants deliberated, self-nominated, and elected a leader whose task performance determined group rewards.

We then simulated matched LLM groups using Gemini 2.5 Flash, GPT 4.1 mini, Claude Haiku 3.5, and Gemma 3. These LLM agents were conditioned on human demographic profiles and task-relevant survey responses. A counterfactual no-demographics (ND) condition was also included for LLMs to isolate the effect of internal identity awareness. We benchmarked models on "alignment" (matching human leader choice) and "optimality" (selecting the best-performing leader, measured by an "optimal leader gap").

Gemini & GPT: Replicating Human Social Patterns

Our findings reveal that when provided with explicit demographic information, models like Gemini and GPT act as "mirrors," reproducing human social patterns, biases included. In the identified condition, human groups elected male leaders 64% of the time, mirroring established gender gaps in leader selection.

Gemini closely replicated this, showing significant alignment (up to 46% exact match with human-elected leaders in identified conditions, Fig. 3) and reproducing both the magnitude and decomposition of the human optimal leader gap (Fig. 4). This includes mirroring male-skewed self-nomination and peer-exclusion patterns. This mirroring property makes these models valuable for accurate social behavior simulation, crucial for mechanism design and predictive modeling where understanding existing human dynamics is key.

Claude: Projecting Idealized Outcomes

In stark contrast, Claude (Haiku 3.5) behaves as a "mask," projecting more meritocratic outcomes. While exhibiting low alignment with human-elected leaders, Claude consistently chose more optimal leaders, achieving an optimal leader gap of just 2% (compared to 14.5% for humans and 10.7% for Gemini in identified conditions).

This suggests that Claude's internal mechanisms may activate corrective behaviors in the presence of visible identity cues, attempting to mitigate human biases and promote more objective decision-making. This "masking" property is beneficial in mediation or intervention settings where the goal is to drive normative or corrective behaviors, rather than merely simulating current human tendencies.

Context-Dependent Alignment: Identified, Pseudonymous, and No Demographics

The presence or absence of identity cues profoundly shapes LLM behavior and human-AI alignment. Under pseudonymity (HP), human gender gaps in leader selection reduced, primarily due to decreased peer-exclusion. However, self-nomination gaps persisted, suggesting internalized priors.

For LLMs, alignment deteriorated as identity cues were removed. Gemini and GPT's agreement with human group leaders significantly weakened under pseudonymity, persisting only when the human-elected leader was male. This indicates reliance on identity scaffolding for simulating human-like social dynamics. The counterfactual no-demographics (ND) condition for LLMs resulted in a complete loss of alignment, underscoring that explicit persona construction with identity cues is essential for effective social simulation. When identity cues were removed, all models defaulted to male-aligned choices, suggesting ingrained gendered priors even without explicit signals.

Redefining Human-AI Alignment for the Future

Our findings underscore a critical tension in human-AI alignment: the difference between simulation alignment (accurately matching human behavior, including biases) and outcome alignment (achieving normatively better, often more meritocratic, results). The choice of LLM and context variables (e.g., identity cue visibility) significantly influence whether AI systems "mirror" or "mask" human biases.

For enterprises, understanding this distinction is vital. If the goal is to accurately model complex social dynamics for predictive analytics or mechanism design, "mirroring" models like Gemini or GPT may be preferred. If the objective is to promote fairer, more optimal outcomes in decision-making, "masking" models like Claude could be more suitable. Future work must quantify and benchmark this distinction, developing dynamic evaluations that capture the nuances of collective reasoning and integrating insights from social psychology, NLP, and computational behavioral science.