Education & HCI
ST-Buddy: Designing and Evaluating a Course-Grounded LLM Chatbot
This paper presents ST-Buddy, an LLM chatbot designed for academic and administrative support in large introductory courses. It addresses student disengagement due to lack of assistance by providing course-specific, context-aware responses. A formative evaluation showed good usability (SUS=77.81), perceived helpfulness, and relevant/understandable response quality. Findings highlight the potential of modular chatbot frameworks for personalized support and lessons learned for future evaluations.
Executive Impact & Key Findings
Insights into the efficacy and user perception of course-grounded LLM chatbots in higher education.
77.81
System Usability Score (SUS)
75%
Students found ST-Buddy useful
50%
Students use AI daily/almost daily
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
LLM Chatbots for Education
Retrieval Augmented Generation (RAG)
User-Centered Design & Evaluation
LLM Chatbots for Education
Large Language Models (LLMs) are increasingly explored for scalable educational support, offering adaptive and personalized interactions. ST-Buddy leverages LLMs to provide course-specific assistance, reducing tutor workload and addressing student queries.
Retrieval Augmented Generation (RAG)
ST-Buddy combines modular LLM-based dialogue with flexible course-grounded knowledge integration using RAG. This enables adaptive, context-aware responses grounded in relevant course material and logistical information, ensuring accuracy and relevance.
User-Centered Design & Evaluation
The design of ST-Buddy emphasizes user-centered processes, data privacy, and a transparent technical architecture. Formative evaluation (n=32) showed good usability and perceived helpfulness, informing future improvements and highlighting key HCI requirements for sustained use.
77.81
Average System Usability Score (SUS) for ST-Buddy, indicating good usability.
Enterprise Process Flow
User Query
→
Bot Routing (FastAPI)
→
Query Rewriting
→
RAG (VectorDB)
→
LLM Call (LangChain/LiteLLM)
→
Response
| Feature | ST-Buddy | General Chatbots |
|---|---|---|
| Knowledge Grounding |
|
|
| Privacy & Data Handling |
|
|
| Support Focus |
|
|
| Evaluation Type |
|
|
Impact in Software Technology Course
ST-Buddy was piloted in an introductory Software Technology course to address students' recurring struggles with information access. It provided scalable learning and organizational support, complementing human assistance. Students reported using AI for explanations, programming support, and exam preparation frequently. The chatbot's ability to provide relevant and understandable responses directly from course material was highly valued, especially when human tutors were unavailable.
Calculate Your Potential ROI
Estimate the efficiency gains and cost savings your institution could achieve with a tailored AI chatbot.
Estimated Annual Savings
$150,000
Annual Hours Reclaimed
10,000
Your Implementation Roadmap
A typical phased approach to integrate ST-Buddy and realize its full potential within your institution.
Phase 1: Discovery & Integration
Initial workshop to define requirements, integrate existing course materials, and set up the RAG pipeline. Duration: 2-4 weeks.
Phase 2: Pilot Deployment & Feedback
Deploy ST-Buddy for a pilot group, collect user feedback through surveys and direct interaction, and iterate on response quality and usability. Duration: 4-6 weeks.
Phase 3: Refinement & Scaling
Implement improvements based on pilot results, expand knowledge base, and prepare for wider deployment across multiple courses. Duration: 6-8 weeks.
Ready to Transform Your Educational Support?
Connect with our AI specialists to explore how ST-Buddy can be customized for your unique academic environment.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat