Medical Research Methodology
Using artificial intelligence for systematic review: the example of elicit
This study evaluates the effectiveness of Elicit, an AI tool, in assisting systematic reviews compared to traditional methods. It assesses Elicit's repeatability, reliability, and accuracy by comparing an AI-assisted search with an independent umbrella review. Findings suggest Elicit is a valuable complementary tool but has limitations, highlighting the need for human oversight and methodological rigor.
Executive Impact: At a Glance
The integration of AI tools like Elicit in systematic reviews presents both efficiency gains and challenges in methodological rigor. Our analysis reveals key performance metrics and areas for careful implementation.
0
Overlap with Traditional Method
0
New Articles Identified by AI
0
Common Articles Identified
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Background
Methods
Results
Discussion
Conclusion
Systematic reviews are rigorous and time-consuming. AI tools, such as Elicit (based on language models like GPT-3), are emerging to assist, identifying relevant papers, summarizing questions, and organizing results. Elicit's unique custom report feature and filter options make it attractive. This research investigates if AI-assisted screening with Elicit adds value compared to traditional methods for an umbrella review on smart living environments for older adults.
The study compared an AI-assisted systematic review using Elicit with an existing umbrella review. The research question, "What is the effectiveness of smart living environments in supporting ageing in place?", was entered into Elicit. Filters for 'systematic review' and publication years (2005-2021) were applied. The search was repeated three times for repeatability assessment. Accuracy was determined by manually assessing Elicit's retrieved articles against the umbrella review's inclusion criteria. Reliability was measured by comparing the number of shared publications between Elicit and the traditional method.
Repeatability tests for Elicit showed varied results across three trials (246, 169, and 172 articles, respectively). After pooling and de-duplication, 241 articles were identified. Accuracy testing concluded with 6 articles included after full-text review using the umbrella review's criteria. Reliability comparison revealed 3 common articles between Elicit and the traditional method, 3 exclusively identified by Elicit, and 17 exclusively by the AI-independent review.
AI research assistants like Elicit are valuable complementary tools for systematic reviews but cannot fully replace traditional methods due to limitations in repeatability and reliability. Elicit identified only 17.6% of studies found by the traditional method but also found 3 new articles. Its accuracy is influenced by research question formulation and its reliance on a single database (Semantic Scholar). The study emphasizes the need for human oversight to maintain methodological rigor and address issues like incorrect referencing by Elicit.
AI tools like Elicit can enhance time efficiency, optimization, and rigor in systematic reviews when combined with human analysis. However, they have limitations in validity, reliability, and accuracy, requiring adherence to principles like transparency and methodological rigor. Future improvements and reporting guidelines (e.g., PRISMA AI) will further enhance their effectiveness.
246
Maximum Articles Retrieved by Elicit in a Single Trial
The highest number of articles Elicit retrieved in one of its three trials for repeatability assessment, indicating variability in search outcomes.
Enterprise Process Flow
Define Research Question
→
AI-Assisted Search (Elicit)
→
Duplicate Removal
→
Title/Abstract Screening
→
Full-Text Review
→
Final Inclusion
| Criterion | Elicit-Assisted Approach | Traditional Manual Screening |
|---|---|---|
| Repeatability | Variable results across trials (246, 169, 172 articles) | High, given consistent methodology |
| Reliability (Overlap) |
|
|
| Accuracy |
|
|
| Efficiency | Faster initial screening and summary generation | Time-consuming, requires extensive manual effort |
| Comprehensiveness | Limited by single database (Semantic Scholar), inconsistent search results | Broader database coverage, more exhaustive |
| Methodological Rigor | Requires human oversight for validity (e.g., referencing) | Established protocols, prone to human bias |
Elicit's Impact on Systematic Review Efficiency
While Elicit demonstrated variability in its search results, its ability to quickly generate summaries and identify potentially relevant articles significantly streamlines the initial phases of a systematic review. This efficiency gain allows researchers to allocate more time to critical analysis and synthesis.
Key Takeaway: AI tools like Elicit are most effective when integrated as assistants to accelerate repetitive tasks, freeing up human experts for nuanced judgment and quality control.
Advanced ROI Calculator
Estimate the potential return on investment for integrating AI into your systematic review processes.
Estimated Annual Savings
$0
Hours Reclaimed Annually
0
Your AI Implementation Roadmap
A structured approach to integrating AI tools like Elicit for maximum impact and minimal disruption.
Phase 1: AI Readiness Assessment
Evaluate existing systematic review workflows, identify bottlenecks, and assess the potential for AI integration. Define clear objectives for AI tool adoption.
Phase 2: Pilot AI Tool Implementation (Elicit)
Integrate Elicit into a small-scale systematic review project. Train researchers on its use, focusing on its strengths in initial search and summary generation while emphasizing human validation.
Phase 3: Performance Validation & Customization
Compare Elicit's outputs (repeatability, reliability, accuracy) against traditional methods. Customize search queries and filters to optimize results for specific research questions. Develop internal guidelines for AI use.
Phase 4: Scaled Deployment with Oversight
Expand Elicit's use to broader systematic review initiatives. Implement robust human oversight mechanisms for critical steps like full-text review and data extraction, ensuring methodological rigor and integrity.
Phase 5: Continuous Improvement & Feedback
Regularly review AI tool performance and researcher feedback. Stay updated on AI advancements and adapt integration strategies, contributing to the development of best practices and reporting standards for AI in systematic reviews.
Ready to Transform Your Research with AI?
Unlock the full potential of AI for systematic reviews. Our experts are ready to design a tailored strategy for your organization.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat