Enterprise AI Analysis

Performance Outcomes and User Acceptance

The functional testing revealed a commendable semantic accuracy rate of 87%. Out of 100 diverse testing scenarios, 87 responses were deemed relevant, 9 fairly relevant, and only 4 not relevant. This demonstrates the chatbot's strong generalization capabilities, interpreting varied linguistic expressions and delivering contextually appropriate answers.

The system's ability to maintain relevance despite variations in sentence construction highlights the effectiveness of combining Q-Learning with Sentence-Transformers for deep semantic understanding.

User Acceptance Testing (UAT) confirmed the system's practical usability. Users provided feedback on dimensions such as satisfaction, perceived ease of use, clarity of responses, and trust in the answers provided by the digital consultation system. The efficient API handling and lightweight Flutter application ensured quick response times, contributing to a positive user experience (see Figure 5 and 6 in the original article for mobile UI example).

Current Limitations & Future Directions

Despite its successes, the Sharia chatbot faces several limitations:

• Static Q-Table: The model does not support continuous learning post-deployment, relying solely on its initial training dataset. This hinders dynamic adaptation to new expressions or emerging issues in Islamic discourse.
• Dataset Dependency: The quality, depth, and representativeness of the dataset heavily influence answer accuracy, especially for complex theological or multi-layered contextual interpretations.
• Q-Learning Scalability: Q-Learning is not ideal for environments with very large or continuous state spaces, which can arise from high-dimensional embeddings.
• Computational Constraints: Real-time performance is affected by the intensive generation of sentence embeddings, network conditions, and backend loading times.
• Lack of Multi-Turn Conversation: The current model does not inherently support context retention across multiple turns, limiting its ability to handle complex follow-up questions.

Future research aims to address these limitations by:

• Dataset Expansion: Incorporating more comprehensive and multilingual Islamic sources.
• Reinforcement Learning from Human Feedback (RLHF): To refine answer selection and enable continuous learning and adaptation.
• Semantic Context Tracking: Implementing mechanisms for multi-round conversations to maintain context.
• Enhanced User Interaction: Integrating speech-based input/output and user profiling for personalized responses.
• Cloud Integration: Leveraging cloud-based knowledge repositories for improved scalability and accessibility.