Enterprise AI Analysis
In the Blink of an AI: Exploring Large Language Models' Capability to Infer Traits From LinkedIn
Large language models (LLMs) are increasingly promoted to practitioners as tools for inferring personality traits from LinkedIn profiles, promising scalable and innovative assessments. Yet, the psychometric foundations of such inferences remain untested. Building on the lens model, we presented 406 LinkedIn profiles to Microsoft Copilot (powered by GPT-4) twice, using single-shot prompting to assess personality (Big Five, narcissism) and intelligence. Inferences showed satisfactory intra-rater reliability for observable traits (up to r=.81), but poor reliability for less visible traits, suggesting unstable inferences (as low as r=.31). Correlations with ground-truth test scores indicate above-chance yet limited convergent validity for intelligence (r=.24), openness (r=.20), and extraversion (r=.20), but not for less visible traits. Analysis of 32 coded LinkedIn cues suggests that this above-chance convergence reflects Copilot drawing on LinkedIn information with some consistency and sensitivity to valid trait signals. While this suggests a rudimentary functional grasp of personality, inferences were undermined by serious flaws, including positivity bias, range restriction, poor discriminant validity, cue overgeneralization, and adverse demographic impacts. By extending the lens model to LLMs as perceivers, we offer a theoretical and empirical foundation for understanding LLM-based trait inferences. Overall, claims that LLMs can validly infer personality from LinkedIn profiles are not just overoptimistic, but potentially harmful—they risk encouraging the adoption of practices that could lead to invalid selection decisions, unfair treatment of applicants, and legal exposure for organizations.
Quantifying the Risks of LLM-Based HR Inferences
Our analysis reveals significant psychometric deficiencies in LLM-based personality inferences from LinkedIn profiles, posing substantial risks for organizations relying on these tools for high-stakes HR decisions.
Reliability for Observable Traits
Reliability for Less Visible Traits
Max Convergent Validity (Intelligence)
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Reliability Concerns
Validity & Bias
Ethical & Legal Implications
Adverse Impact
While Copilot demonstrated acceptable intra-rater reliability for most traits, it was unacceptable for narcissism and poor for agreeableness and neuroticism, suggesting unstable inferences that cannot be relied upon. Analysis of cue utilization revealed that these less reliably inferred traits were also associated with lower average cue utilization, indicating that Copilot applied less consistent rules when inferring them. This inconsistency may arise from the lack of clear, visible cues for traits like agreeableness and neuroticism on LinkedIn.
Lowest Intra-Rater Reliability (Neuroticism)
LLMs showed limited convergent validity (max r=.24 for intelligence) and poor discriminant validity, indicating they don't estimate discrete traits but rather produce broad, non-specific individual difference profiles. Systematic biases include positivity bias, range restriction, and cue overgeneralization, leading to inflated ratings for reasons unrelated to actual traits.
Enterprise Process Flow
Input LinkedIn Profiles
→
Copilot Trait Inference
→
Limited Convergent Validity
→
Poor Discriminant Validity
→
Positivity Bias & Range Restriction
→
Adverse Impact Risk
Copilot's reliance on invalid cues and overgeneralization raises direct legal risks under EU and U.S. law. AI-based selection systems must be comprehensible, traceable, and explainable. The lack of these properties in Copilot's inferences makes full compliance unlikely, posing risks of discrimination and legal exposure for organizations.
| Requirement | LLM Performance (Copilot) |
|---|---|
| Comprehensible Cues |
|
| Traceable Decisions |
|
| Explainable Outcomes |
|
Copilot exhibited systematic differences in favorability (directional bias) and absolute prediction error (precision bias) across gender and age groups. For instance, women were assigned higher openness scores despite no actual differences, and older individuals were incorrectly rated as more conscientious/extraverted/narcissistic and less neurotic. These disparities, even if small, can lead to entrenched inequities and reduced workforce diversity.
Case Study: Gender & Age Biases in LLM Inferences
Scenario: A large enterprise adopted an LLM for initial candidate screening based on LinkedIn profiles. The LLM consistently rated female candidates higher in 'Openness' than male candidates, despite no real differences in actual test scores. Similarly, older candidates were frequently assigned inflated scores for 'Conscientiousness' and 'Extraversion' but also 'Narcissism', and lower for 'Neuroticism', leading to an unfair advantage for older applicants in some dimensions and disadvantage in others.
Outcome: This led to subtle, yet systemic, demographic disparities in candidate evaluations over time. The organization faced increased risks of legal challenges under anti-discrimination laws and observed a reduction in the diversity of its applicant pool for certain roles. The 'positivity bias' of the LLM amplified these issues, masking true underlying trait levels and skewing hiring recommendations.
Recommendation: Implement robust validation studies for any AI-based assessment tools, focusing on psychometric properties like reliability, validity, and fairness. Conduct regular bias audits using diverse datasets. Avoid using LLMs for high-stakes hiring decisions without specialized training and rigorous testing.
Quantify Your AI Efficiency Gains
Estimate the potential annual cost savings and hours reclaimed by implementing smarter AI-driven processes in your organization.
Estimated Annual Savings
$0
Hours Reclaimed Annually
0
Your AI Implementation Roadmap
A clear path to integrating advanced AI into your enterprise, tailored to maximize impact and minimize disruption.
Discovery & Strategy
Initial consultation to understand your business needs, identify AI opportunities, and define strategic objectives. Data assessment and feasibility study.
Pilot Program Design
Develop a tailored pilot project, selecting key use cases and defining success metrics. Set up necessary infrastructure and data pipelines.
AI Solution Development
Build and train custom AI models. Integrate solutions with existing systems. Rigorous testing and validation for performance and bias.
Deployment & Scaling
Roll out AI solutions across your organization. Provide comprehensive training for your team. Monitor performance and iterate for continuous improvement.
Ready to Transform Your Enterprise with AI?
Partner with Own Your AI to leverage cutting-edge research and tailored solutions for unparalleled efficiency and strategic advantage.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat