Enterprise AI Analysis

Qualitative Design for Nuanced Insights

This exploratory study employs a qualitative approach to understand the nuanced dynamics of GenAI-supported learning. It focuses on process, perception, and interaction rather than causal effects or performance metrics, aiming to surface design-relevant patterns for future controlled evaluations.

The research addressed three key questions:

• ✓ RQ1: When and how should the system intervene to maintain perceived usefulness and positive engagement while preserving learner autonomy?
• ✓ RQ2: Which prompt types (guiding questions, concept checks, strategy hints) are perceived as most helpful at moments of impasse?
• ✓ RQ3: How do learners use post-run, holistic feedback to progress through the experiential learning cycle?

Empirical Patterns from Early GenAI Support

The qualitative analysis, based on system logs, field observations, and post-activity reflections from 9 novice programmers, identified three key patterns:

• ✓ Supportive Interventions: Short-delay interventions were generally perceived as supportive rather than disruptive.
• ✓ Effective Prompt Types: Open-ended guiding questions were more effective than direct hints in stimulating reflection and self-correction.
• ✓ Consolidation via Feedback: Post-run feedback encouraged sense-making and consolidation of concepts within experiential learning cycles.

Overall, the findings suggest that minimally intrusive, question-led GenAI support can sustain perceived usefulness and engagement while maintaining learner autonomy.

Strategic Roadmap for Scalable AI Tutoring

Future work will address current limitations and expand the research scope to build a robust, evidence-based GenAI tutoring system:

• ✓ Larger & Diverse Cohorts: Run studies with more students across a wider range of tasks and programming features.
• ✓ Controlled Comparisons: Evaluate GenAI tutor effectiveness against non-AI hint mechanisms (e.g., static examples, rule-based prompts) and conventional clinic practice.
• ✓ Quantitative Performance Measures: Incorporate task completion rates, error counts, and independent code quality assessments.
• ✓ Adaptive Intervention: Implement a learner-initiated "hands up" mechanism to reduce intrusiveness and gather richer contextual data for refining adaptive, automatic triggers.