Enterprise AI Analysis

This paper investigated the effectiveness of Large Language Models (LLMs) in automatically detecting Insufficient Effort Responding (IER) in open-ended survey questions. Two primary pipelines were evaluated: feature extraction using open-source embedding models with classifiers, and text generation using GPT-40-mini. While both approaches performed well on the majority class (valid responses), they struggled with the minority class (IER) due to severe class imbalance. The study highlights the potential of LLMs in HCI research for data quality, but emphasizes the need for human-in-the-loop workflows and advanced model engineering to overcome current limitations.

The study utilized a dataset of 1,551 open-ended responses from HCI player experience research. Participants were instructed to describe a digital game in at least 50 words, ensuring genuine engagement. A human rater manually classified responses based on adherence to instructions, memory of the game, and description validity, establishing a "ground truth" for IER detection. This contextualized dataset provided an ideal test case for evaluating LLM-based approaches, moving beyond superficial anomaly detection.

Comparative Performance of IER Detection Pipelines

Evaluation of feature extraction (embedding models with classifiers) versus text generation (GPT-40-mini) performance, highlighting the trade-offs in precision, recall, and overall accuracy for detecting insufficient effort responding (IER) in open-ended survey questions.

Integrating LLMs into Human-in-the-Loop Workflows

Given the LLMs' tendency to overpredict IER, they can be productively integrated into human-in-the-loop workflows to enhance research rigor and efficiency. Two main approaches include full human-in-the-loop annotation, where LLMs act as independent annotators for inter-rater reliability checks, helping to identify potential IER cases missed by human researchers. Alternatively, semi-automatic annotation allows LLMs to pre-screen responses, flagging likely IER cases for human review, thereby reducing workload. However, the semi-automatic approach requires further validation studies to ensure quality control, making the full human-in-the-loop route safer for rigorous research in its current state.

• LLMs can act as independent annotators, identifying discrepancies in human labeling.
• Pre-screening by LLMs can reduce human workload in large datasets.
• Rigorous validation is crucial for semi-automatic annotation workflows.
• Current recommendation: Full human-in-the-loop for maximum rigor and transparency.

Key Takeaways and Engineering Advancements for HCI Researchers

This exploratory study indicates that off-the-shelf LLMs do not yet offer a robust, standalone solution for automated IER detection in open-ended survey data. While embedding-based classifiers are useful for exploratory analysis and are inexpensive, they tend to miss IER. Text generation approaches show higher sensitivity but significantly overclassify IER. Therefore, current research practice should prioritize human-in-the-loop annotation to ensure rigor. Future advancements will likely require more intricate workflows, improved handling of class imbalance, and fine-tuning of large language models on curated, domain-specific datasets to improve reliability and scalability of IER detection in HCI research.

• Off-the-shelf LLMs are not a robust standalone solution for IER detection.
• Embedding models: inexpensive, good for exploration, but may miss IER.
• Text generation models: sensitive but prone to overclassification.
• Prioritize human-in-the-loop annotation for rigorous research.
• Future work: intricate workflows, class imbalance strategies, fine-tuning LLMs on curated datasets.