Enterprise AI Analysis
Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review
This analysis of CHI papers (2020-2024) reveals the extensive integration of Large Language Models (LLMs) across diverse HCI domains. It identifies key application areas, LLM roles in research, contribution types, and prevalent limitations articulated by authors, particularly concerning validity and reproducibility.
Executive Impact Summary
Our analysis reveals the rapid growth and diverse applications of Large Language Models within HCI research, highlighting critical trends for enterprise AI strategy.
0
LLM-Related Papers Analyzed
0
Key LLM Roles Identified
0
Application Domains
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Application Domains
LLMs are revolutionizing how HCI researchers engage with various fields, from communication to healthcare.
- Communication & Writing: Most studied, enabling AI-mediated communication and content generation.
- Augmenting Capabilities: Enhancing human performance in tasks like video conferencing and business analysis.
- Education: Improving learning experiences and pedagogical methods.
- Responsible Computing: Addressing ethical and societal implications, focusing on fairness and privacy.
- Programming: Automating and improving software development, including code generation and prompt engineering.
- Reliability & Validity of LLMs: Evaluating and improving LLM outputs, identifying biases and hallucinations.
- Well-being & Health: Managing health conditions and assisting healthcare providers.
- Design: Facilitating design processes for UI, landscape, and multimodal applications.
- Accessibility & Aging: Focusing on people with disabilities and older adults.
- Creativity: Supporting creative processes and tools, scrutinizing LLM 'creativity'.
LLM Roles in HCI Projects
LLMs take on distinct roles in HCI research, from system engines to objects of study.
- As System Engines: Core elements in prototypes, algorithms, and programming frameworks, generating content or processing information.
- As Research Tools: Performing tasks like data collection, analysis, or writing traditionally done by researchers.
- As Participants & Users: Simulating human responses and behaviors, or acting as users in interactions.
- As Objects of Study: Evaluating LLMs' underlying mechanisms, properties, and performance (e.g., audits).
- Users' Perceptions of LLMs: Studying how users perceive and interact with LLM-powered tools (e.g., ChatGPT).
Limitations & Risks
Authors identified common limitations and risks, spanning performance, validity, and societal consequences.
- LLM Performance: Concerns about output quality, bias, non-determinism, and hallucination.
- Research Validity: Issues with internal/external validity due to limited samples, context dependency, and prompt sensitivity.
- Resource Limitation: Computational and financial costs, lack of evaluation standards.
- Risks to Society: Potential negative long-term outcomes like economic harms, representational harms, and misinformation.
84.98%
of papers studied closed GPT-family models
Enterprise Process Flow
Query ACM DL (4077 papers)
→
Keyword Search (152 papers)
→
Robustness Check (+1 paper)
→
153 Papers for Analysis
| Feature | LLM as Research Tool | Human Researcher |
|---|---|---|
| Speed |
|
|
| Nuance & Context |
|
|
| Reproducibility |
|
|
| Cost |
|
|
Case Study: LLMs in Education
Researchers explored LLMs' potential to enhance learning and pedagogical methods. Findings highlight both excitement about personalized feedback and concerns regarding student agency, bias, and misinformation. This requires careful integration and robust evaluation protocols.
- Students' interactions with LLMs (e.g., ChatGPT) as learning aids.
- LLMs generating teaching materials and personalized feedback.
- Concerns: student agency, bias, misinformation, need for responsible AI.
Calculate Your Potential AI ROI
Estimate the efficiency gains and cost savings your enterprise could achieve by strategically implementing AI.
Your AI Implementation Roadmap
A strategic phased approach to integrate LLMs into your enterprise, ensuring scientific rigor and responsible design.
Phase 1: Discovery & Assessment
Identify high-impact use cases and conduct a feasibility study, evaluating existing LLM solutions.
Phase 2: Pilot & Prototype
Develop a minimal viable product (MVP) with chosen LLM, focusing on a specific business process.
Phase 3: Integration & Scaling
Integrate the LLM solution into existing workflows and scale across the enterprise, with continuous monitoring.
Phase 4: Optimization & Governance
Establish long-term governance, monitor performance, and refine models for continuous improvement.
Ready to Transform Your Enterprise with AI?
Book a personalized consultation to explore how these insights can drive your strategic AI initiatives.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat