Skeleton-Cache: A Training-Free, Adaptive Framework

We introduce Skeleton-Cache, the first training-free test-time adaptation framework specifically designed for SZAR. It significantly improves model generalization to unseen actions during inference without requiring any additional training or access to training data.

Skeleton-Cache reformulates inference as a lightweight retrieval process over a non-parametric cache. This cache stores structured skeleton representations, combining both global and fine-grained local descriptors. Crucially, it leverages the semantic reasoning capabilities of Large Language Models (LLMs) to assign class-specific importance weights, guiding the fusion of descriptor-wise predictions and enabling dynamic adaptation to novel actions.

How Skeleton-Cache Works

The core of Skeleton-Cache involves three key processes: Cache Construction and Update, Descriptor-wise Prediction, and LLM-Guided Prior Fusion.

• Cache Construction: The cache stores structured feature representations (cache keys) for skeleton sequences, along with predicted labels and confidence scores. These keys combine global descriptors with fine-grained spatial (body parts) and temporal (motion phases) descriptors.
• Dynamic Cache Update: During inference, high-confidence entries from test samples are selectively inserted into class-specific blocks in the cache, ensuring a compact yet representative memory that continuously adapts to novel inputs without gradients.
• Descriptor-wise Prediction: For each incoming test sample, its descriptors are compared with all cached entries to compute similarity scores, generating descriptor-wise class logits.
• LLM-Guided Fusion: Large Language Models are queried to provide class-specific importance weights for global, spatial, and temporal descriptors. These weights guide an adaptive, weighted fusion of descriptor-wise predictions, enhancing classification accuracy without extra training.