AI in Accessibility

Developing an LLM-Powered Virtual Guide

This study built upon a previously developed AI guide prototype, focusing on key "personas" (dog, robot, human) to offer varied guidance experiences. The guide's core functionalities included providing real-time visual descriptions, assisting with avatar movement to specified locations, and placing spatialized audio beacons for orientation.

The system leveraged OpenAI's Whisper for Speech-to-Text and GPT-4 for generating responses. A key enhancement included a refined prompt structure specifically tailored for BLV users, incorporating best practices like using distances, cardinal directions, and limiting response length to prevent overwhelming users. This ensures the guide provides contextual and relevant information in a dynamic VR environment.

Dynamic User Behavior: From Tool to Companion

Participants exhibited significantly different interaction patterns with the AI guide based on the social context. When engaging with the guide alone, users primarily treated it as a utilitarian tool, employing direct commands and keeping interactions focused on their visual needs. This reflects a task-oriented approach to problem-solving.

However, when confederates (posing as other users) joined the VR environment, participants' behavior shifted. They adopted a more companionable and role-playing approach, giving the guide nicknames, using gendered pronouns, rationalizing its mistakes based on its persona (e.g., a dog "going to sleep"), and even encouraging confederates to interact with the guide. This highlights the guide's potential for fostering social connection beyond mere utility.

Assessing Guide Performance and User Experience

The AI guide proved effective in assisting participants to explore virtual parks and accurately share information with confederates, demonstrating its utility for navigation and information retrieval. Most participants successfully guided groups to multiple landmarks. The guide achieved an overall query accuracy of 63.2%, with some inaccuracies stemming from misinterpreted accents or incomplete queries.

Response times varied, with visual questions taking the longest (11.2 seconds). While participants found the guide useful and generally enjoyable (mean 4.1/5 joy of use), there were mixed opinions on usability and social comfort. Some BLV users preferred to control the disclosure of their disability, leading to discomfort with the guide's visible presence.

Future-Proofing AI Guides for Enhanced Accessibility

To maximize the utility and user experience of future AI guides for BLV users, several design recommendations emerge from this study:

• Foster Emotional Connections: Incorporate interactive capabilities that support and respond to user role-playing and social engagement.
• Encourage Advanced Usage: Implement features that explicitly demonstrate the guide's capacity for complex tasks, offering proactive visual details and checking on user well-being.
• Align with User Roles: Provide explicit failsafe mechanisms, such as a one-button "return to safe location," rather than requiring voice commands.
• Reflect Capabilities in Persona: Ensure the guide's visual and verbal persona accurately reflects its capabilities and experience level (e.g., a "puppy" guide for less experienced users).
• Actively Assist Prompting: Implement proactive teaching mechanisms to help users develop better prompt engineering skills for more effective queries.
• Address Latency and Accuracy: Continuously improve STT and LLM integration for faster, more reliable responses and better context handling.