Enterprise AI Analysis: Healthcare
A comparative analysis of embedded chatbot models and ChatGPT-4 for answering orthodontic treatment queries
Authored by Rizwan Khalil, Laiba Amin, Rashna Hoshang Sukhia & Mubassar Fida
Published in Scientific Reports (2026) | DOI: 10.1038/s41598-026-39263-3
Executive Impact
Understanding the real-world implications of specialized AI in patient care.
0
Embedded Chatbot I-CVI
0
ChatGPT-4 I-CVI
0
Embedded Chatbot S-CVI (Relevance)
0
Statistical Significance (p-value)
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Key Findings
This study compared an embedded chatbot model with ChatGPT-4 for answering orthodontic treatment queries. The embedded model, utilizing domain-specific embedding and prompt engineering with Claude 3.7 Sonnet, demonstrated higher content validity and better scores in accuracy and clarity. While numerical trends favored the embedded model, statistical significance was not reached (p = 0.173). The findings underscore the growing role of tailored AI in enhancing patient communication and clinical support within orthodontics, suggesting advantages of domain-specific augmentation over generic LLMs.
Context & Problem
Orthodontics, a specialized branch of dentistry, relies heavily on effective patient communication for treatment information. Traditional methods have limitations in accessibility, immediacy, and scalability. The advent of AI and large language models (LLMs) like ChatGPT presents an opportunity to overcome these challenges by providing continuous support and personalized information. However, a thorough evaluation of their effectiveness in domain-specific contexts like orthodontics is crucial, especially when comparing tailored models against generalized ones.
Approach & Data
A cross-sectional comparative study was conducted, comparing an embedded chatbot (built with Anthropic's Claude 3.7 Sonnet via Google Colab, leveraging Retrieval-Augmented Generation (RAG) with curated orthodontic reference documents) against ChatGPT-4 (OpenAI, USA). A questionnaire of 30 real-world orthodontic queries was developed and validated by experts. Six orthodontic consultants (>=5 years experience) evaluated responses from both models using a 5-point Likert scale across four dimensions: accuracy, clarity, relevance, and up-to-date knowledge. Content validity was assessed using item-level (I-CVI) and scale-level (S-CVI) indices. Statistical analysis involved Wilcoxon signed-Rank test and Shapiro-Wilk test, with inter-rater reliability assessed via modified kappa.
Embedded Chatbot Model Development
Build the embedded chatbot model utilizing Anthropic's Claude 3.7 Sonnet through Google Colab
→
Access the model via Application Programming Interface (API) integration
→
Employ prompt engineering to define a strict role for the chatbot in addressing orthodontic treatment-related queries
→
Upload a contextual knowledge base of curated orthodontic care instructions as a reference document
→
Create relevant embeddings using Retrieval-Augmented Generation (RAG) techniques and store them in a vector database
→
Set the temperature to a lower value to reduce hallucination and ensure consistent, evidence-based outputsts
Response Evaluation Study Design
A questionnaire comprising 30 real-world orthodontic treatment-related queries was constructed.
→
Questions validated by domain-specific experts
→
Responses generated via embedded chatbot model and OpenAI's ChatGPT-4, anonymized as 'A' and 'B'
→
Responses disseminated to evaluators on Google Forms and assessed using a 5-point Likert scale: accuracy, clarity, up-to-date knowledge, and relevance
Item-Level Content Validity (I-CVI) Performance
| Variable | Embedded chatbot (%) | ChatGPT-4 (%) |
|---|---|---|
| Accuracy | 23 (76.66) | 16 (53.33) |
| Clarity | 21 (70.00) | 14 (46.66) |
| Relevance | 26 (86.66) | 14 (46.66) |
| Up-to-date knowledge | 19 (63.33) | 17 (56.66) |
| Overall | 89 (74.16) | 61 (50.83) |
Scale-Level Content Validity (S-CVI) Performance
| Variable | Embedded chatbot | ChatGPT-4 |
|---|---|---|
| Accuracy | 0.811 | 0.711 |
| Clarity | 0.794 | 0.728 |
| Relevance | 0.844 | 0.739 |
| Up-to-date knowledge | 0.778 | 0.644 |
Median (IQR) Scores & Statistical Significance
| Variable | Embedded chatbot Median (IQR) | ChatGPT-4 Median (IQR) | p-value |
|---|---|---|---|
| Accuracy | 4.01 (2.84, 4.68) | 3.71 (3.06, 3.97) | 0.116 |
| Clarity | 3.96 (2.99, 4.63) | 3.50 (2.99, 3.99) | 0.115 |
| Relevance | 4.03 (2.96, 4.65) | 3.66 (3.10, 4.04) | 0.173 |
| Up-to-date knowledge | 4.01 (2.83, 4.67) | 3.61 (3.05, 3.87) | 0.138 |
| Overall content validity | 4.00 (2.90, 4.66) | 3.60 (3.07, 3.97) | 0.173 |
Comparative Responses: Embedded Chatbot vs. ChatGPT-4
Question #1: Will I experience pain or discomfort during braces treatment?
Embedded Chatbot: Yes, you may experience some discomfort or pain during the initial adjustment of your fixed brace. This is normal and should go away within a few days. If you experience any discomfort, take what you normally have for a headache, and be sure to ask your orthodontist for advice.
ChatGPT-4: Yes, it's common to feel some discomfort, especially after braces are first applied or adjusted. This usually lasts a few days and can be managed with over-the-counter pain relievers and soft foods.
Question #2: Do braces affect my singing or speaking?
Embedded Chatbot: No, braces do not affect your singing or speaking ability. Your speech will be different at first, but it will return to normal within a couple of days. Practice speaking with the brace in place, e.g., read out aloud at home on your own. In this way, your speech will return to normal within a couple of days.
ChatGPT-4: Possibly at first. You might notice slight changes, especially if you sing or speak professionally. However, most people adjust quickly, and any initial difficulty usually fades within a couple of weeks.
Question #3: Will brace treatment help with jaw pain or TMJ?
Embedded Chatbot: Brace treatment is designed to help improve the alignment of your teeth and jaws. It is not a remedy for jaw pain or TMJ. Braces are not designed to treat these conditions. However, they can help in the short term with the improvement of your jaw and jaw joint alignment, which can help with pain relief and prevent further damage.
ChatGPT-4: In some cases, yes. Correcting a misaligned bite may relieve TMJ-related symptoms. However, orthodontics isn't a guaranteed cure for TMJ disorders and should be evaluated on a case-by-case basis.
Strategic Implications
The study concludes that domain-specific embedded chatbot models, leveraging prompt engineering and Retrieval-Augmented Generation (RAG) with curated knowledge bases, can offer improved clarity and validity for specialized queries in healthcare, outperforming generic LLMs like ChatGPT-4 in certain metrics, though not always with statistical significance. These findings highlight the potential for tailored AI tools to significantly enhance patient communication and clinical support in orthodontics, by providing more accurate, relevant, and context-aware information.
Calculate Your Potential AI ROI
Estimate the efficiency gains and cost savings for your enterprise by integrating specialized AI solutions.
Estimated Annual Savings
$0
Annual Hours Reclaimed
0
Your AI Implementation Roadmap
A phased approach to integrate specialized AI into your enterprise, maximizing impact and minimizing disruption.
Phase 1: Discovery & Strategy
Conduct a comprehensive assessment of existing workflows, identify key pain points, and define strategic objectives for AI integration. This phase includes data readiness assessment and initial solution design.
Phase 2: Pilot & Proof-of-Concept
Develop and deploy a pilot AI solution in a controlled environment. Validate performance against defined metrics, gather feedback, and refine the model based on real-world usage and expert evaluation.
Phase 3: Scaled Deployment & Integration
Roll out the refined AI solution across relevant departments, ensuring seamless integration with existing enterprise systems. Provide comprehensive training for end-users and establish monitoring protocols.
Phase 4: Optimization & Expansion
Continuously monitor AI performance, gather user feedback, and implement iterative improvements. Explore opportunities to expand AI capabilities to new use cases and departments, fostering ongoing innovation.
Ready to Transform Your Enterprise with AI?
Our experts are ready to guide you through the complexities of AI adoption, from strategy to successful implementation. Book a free, no-obligation consultation to discuss your specific needs.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat