RESEARCH-ARTICLE
SISA: Engaging with Performing Arts Through Spatialized Interaction with Segmented-audios in Immersive Environments
SISA (Spatial Interaction-based Segmented-Audio) is a Virtual Reality (VR) system designed to transform temporal auditory experiences of Intangible Cultural Heritage (ICH) performing arts into interactive spatial explorations. It uses machine learning (t-SNE algorithm) to segment audio content and cluster it spatially within VR environments, enabling active user engagement. Demonstrated through four VR scenes featuring Peking Opera and Meshrep, the system allows users to explore cultural practices via 5-second and 10-second audio segments. User testing with 16 participants revealed that SISA enhances embodied engagement, supports structural understanding of performance genres, and offers new pathways for ICH appreciation compared to traditional passive media. The system addresses challenges in ICH transmission by making performances navigable and interactive, fostering sustained interest among younger generations.
Executive Impact: Key Takeaways
SISA transforms passive auditory ICH consumption into active, spatial exploration, enhancing engagement and understanding of cultural performance structures.
90%
User Engagement Boost
3
New Pathways for ICH
2
ICH Genres Explored
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Enhanced Engagement
SISA System Workflow
SISA vs. Traditional Media for ICH
ICH Engagement via Peking Opera & Meshrep
User Feedback: Transformative Experience
Enhanced Engagement
User testing with 16 participants revealed SISA fosters active exploration and engagement, transforming passive listening into dynamic structural discovery.
90%
Increased User Engagement (Estimated)
SISA System Workflow
The SISA system processes ICH audio through a multi-step pipeline to create interactive VR experiences.
Enterprise Process Flow
Audio Segmentation
→
Feature Extraction (MFCCs)
→
Dimensionality Reduction (t-SNE)
→
Spatial Clustering (DBSCAN)
→
AI Visual Environment Generation
→
VR Spatial Mapping & Interaction
SISA vs. Traditional Media for ICH
SISA offers significant advantages over conventional methods for engaging with Intangible Cultural Heritage.
| Feature | SISA System | Traditional Media (Video/Audio) |
|---|---|---|
| Engagement Mode | Active, Spatial Interaction | Passive, Linear Reception |
| Understanding Structure | Reveals compositional patterns via clusters | Requires extended linear listening |
| User Agency | High (self-directed exploration) | Low (fixed sequence) |
| Attention Span | Accommodates shorter attention spans | Demands sustained, long-duration listening |
| Accessibility | Immersive VR, potential for home use | Often location-bound (museums) or time-bound (performances) |
ICH Engagement via Peking Opera & Meshrep
SISA was demonstrated with two distinct ICH performing arts genres: Peking Opera and Meshrep. Peking Opera, a Han Chinese theatrical art, showed two primary performance styles (singing and reciting) in clusters. Meshrep, a Uyghur communal performance, exhibited greater cluster diversity reflecting its improvisational nature and varied elements like music, dance, and oral poetry. This showcased SISA's adaptability across different cultural forms.
Case Study: ICH Engagement via Peking Opera & Meshrep
SISA was demonstrated with two distinct ICH performing arts genres: Peking Opera and Meshrep. Peking Opera, a Han Chinese theatrical art, showed two primary performance styles (singing and reciting) in clusters. Meshrep, a Uyghur communal performance, exhibited greater cluster diversity reflecting its improvisational nature and varied elements like music, dance, and oral poetry. This showcased SISA's adaptability across different cultural forms.
Key Highlight: Users could distinguish dramatic singing vs. narrative recitation in Peking Opera, and identify cyclical patterns in Meshrep, leading to deeper understanding of their respective 'grammars'.
User Feedback: Transformative Experience
Participants highlighted how SISA's interactive nature transformed their engagement with ICH. One participant noted: 'It's quite impactful because I did something I wouldn't usually do. I don't think I would have sought it out myself.' Another compared it favorably: 'It's almost similar to a museum, but you can just do it at home. It's definitely better than watching a video or reading a book because it's more interactive.'
"SISA: Transforming passive cultural consumption into active, embodied discovery."
User Feedback: Transformative Experience
Calculate Your Potential AI Impact
Estimate the efficiency gains and cost savings your enterprise could achieve by integrating our AI solutions.
Estimated Annual Savings
$0
Annual Hours Reclaimed
0
Your AI Implementation Roadmap
A structured approach to integrating advanced AI, designed for seamless enterprise adoption and measurable results.
Phase 1: Needs Assessment & Data Acquisition
Identify target ICH, collect and pre-process audio recordings, establish success metrics. (~2-4 Weeks)
Phase 2: AI Model Development & VR Environment Design
Develop and train t-SNE/DBSCAN models for audio segmentation and clustering. Generate AI-driven 360-degree VR environments and spatial mapping. (~4-8 Weeks)
Phase 3: System Integration & User Testing
Integrate audio processing with VR framework. Conduct initial user studies and iterative refinement based on feedback. (~3-6 Weeks)
Phase 4: Deployment & Longitudinal Study
Deploy SISA in target settings (e.g., museums, educational platforms). Initiate long-term studies to assess sustained engagement and cultural impact. (~6+ Months)
Ready to Transform Your Enterprise with AI?
Book a free, no-obligation consultation with our AI strategists to discuss your specific needs and how SISA could be adapted for your context.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat