Healthcare
PVminerLLM: Structured Extraction of Patient Voice from Patient-Generated Text using Large Language Models
This paper introduces PVminerLLM, a novel supervised fine-tuned large language model framework for structured extraction of patient voice from patient-generated text. It addresses the critical need for structured patient data to improve patient-centered outcomes and health equity. By leveraging large language models (LLMs) and a carefully designed annotation schema, PVminerLLM achieves high F1 scores across Code prediction (83.82%), Sub-code prediction (80.74%), and Span extraction (87.03%). Notably, it demonstrates strong performance even with smaller models after fine-tuning, making scalable and reliable patient voice extraction feasible without extreme model scale. The framework overcomes limitations of prompt-based approaches by enforcing schema-valid outputs and improving accuracy for nuanced and less frequent patient voice signals, offering significant implications for healthcare delivery and research.
Executive Impact: Key Takeaways
PVminerLLM is set to revolutionize how healthcare organizations analyze patient feedback, leading to measurable improvements in care quality and operational efficiency.
01
Structured Patient Voice Extraction
02
LLM Adaptation Superiority
03
Scalability & Accessibility
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Overview
Methodology
Results
Implications
PVminerLLM: Revolutionizing Patient Voice Extraction
PVminerLLM addresses a critical gap in patient-centered outcomes research and clinical quality improvement: the scarcity of structured patient voice data. Patient-generated texts, such as messages and survey responses, contain invaluable insights into lived experiences, social circumstances, and care engagement (Section 1). However, these signals are rarely in a structured format, limiting their utility at scale. PVminerLLM introduces a novel supervised fine-tuned Large Language Model (LLM) framework specifically designed to extract hierarchical labels (Codes, Sub-codes) and precise evidence Spans from this unstructured text, formalizing patient voice annotation as a schema-constrained structured prediction task (Abstract).
Framework Design and Training Innovations
The PVminer framework formalizes patient voice annotation into a schema-constrained structured prediction task. It defines 8 major Codes with associated Sub-codes to capture high-level communicative or social functions and finer-grained intents or contexts, respectively. The task requires extracting these hierarchical labels along with grounding text Spans (Section 2). The dataset comprises 1,137 patient- and provider-authored messages from diverse healthcare settings, totaling over 46,000 word tokens, annotated using an iterative protocol and eHOST platform (Section 3, Fig. 1). To adapt LLMs for this task, PVminerLLM employs supervised fine-tuning using parameter-efficient QLORA adapters. This method trains models to generate schema-valid JSON outputs by optimizing a masked likelihood objective, ensuring focus on output generation while maintaining computational feasibility (Section 5).
Unprecedented Performance in Patient Voice Extraction
PVminerLLM demonstrates substantial performance improvements over prompt-based baselines across all prediction targets under a zero-shot setting. Supervised fine-tuning significantly boosts F1 scores: up to 83.82% for Code prediction, 80.74% for Sub-code prediction, and 87.03% for evidence Span extraction (Tables 4, 5, 6). The framework's fine-tuned models consistently outperform prompt-only inference, which often yields poorly structured or incomplete outputs due to limitations in schema adherence (Section 6.3, 6.4). Importantly, PVminerLLM achieves strong performance even with smaller models, reducing the dependency on extreme model scale for reliable patient voice extraction (Abstract, Section 7.1).
Transforming Patient-Centered Care at Scale
The ability to accurately extract patient voice domains has profound clinical and social implications. It enables healthcare teams to recognize patients' social and emotional challenges, leading to more informed care plans and targeted interventions (Section 7.2). By providing structured access to information about social determinants of health (SDOH), PVminerLLM helps identify patterns across large populations that would otherwise be overlooked, supporting health equity and patient-centered research. The framework's scalability, even with smaller models, makes it accessible to various healthcare settings. Future work aims to enhance practical deployment by exploring multi-agent inference frameworks and advanced alignment methods for even greater reliability (Section 7.3).
87.03%
F1 for Evidence Span Extraction
Enterprise Process Flow
Codebook Development
→
Annotation
→
Prompt Engineering
→
Supervised Fine-tuning
| Feature | PVminer (Ours) |
|---|---|
| Relational/ Socio-Emotional |
|
| Bidirectional Interaction |
|
| Supports Multi-Label Coding |
|
| Tailored for Secure Messaging |
|
Impact on Patient Voice Domains
PVminerLLM substantially improves identification across nearly all patient voice domains. For example, in PartnershipPatient, F1 scores increased from 83.82% (two-shot) to 88.41% (SFT). Similarly, SDOH domain F1 improved from 60.22% to 89.26%, indicating better detection of socio-economic concerns. This highlights the effectiveness of supervised fine-tuning in capturing complex and nuanced patient voices.
Calculate Your Potential AI ROI
Estimate the efficiency gains and cost savings your enterprise could achieve by implementing our AI solutions.
Annual Savings
Hours Reclaimed Annually
Your AI Implementation Roadmap
A clear, phased approach to integrating advanced AI into your enterprise operations for maximum impact.
Phase 1: Discovery & Strategy
Initial consultations and workshops to understand your specific needs, challenges, and strategic objectives. We define project scope, key performance indicators, and a tailored AI strategy.
Phase 2: Pilot & Proof-of-Concept
Develop and deploy a small-scale pilot project to demonstrate the AI solution's effectiveness in a controlled environment, gathering critical feedback for refinement.
Phase 3: Full-Scale Integration
Roll out the AI solution across your enterprise, ensuring seamless integration with existing systems, comprehensive training for your teams, and ongoing support.
Phase 4: Optimization & Scaling
Continuous monitoring, performance tuning, and identification of new opportunities to expand AI capabilities and deliver increasing value across your organization.
Ready to Transform Your Enterprise with AI?
Book a complimentary strategy session with our AI experts to explore how PVminerLLM and other custom solutions can drive your organization forward.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat