AI-POWERED SKELETON ANALYSIS

Skeleton-to-Image Encoding (S2I)

S2I is a novel representation method that reformats skeleton sequences into dense, image-like data compatible with vision models. It addresses the fundamental challenge of applying vision models to sparse 3D skeleton data by converting 3D joint coordinates (x, y, z) directly to RGB channels, transforming motion patterns into pseudo-images.

The process involves: 1. Joint Partitioning: Skeleton joints are partitioned into five semantic body parts (torso, left arm, right arm, left leg, right leg) to ensure semantic consistency. 2. Joint Reordering: Joints within each part are sorted top-down based on physical positions, following kinematic chains (e.g., left hip → left knee → left ankle → left foot). 3. Temporal Stacking: Reordered joints across the temporal dimension (T frames) are stacked to form a spatial-temporal feature map of size T × J. 4. Resizing & Interpolation: The representation is resized to standard image input dimensions (e.g., 224 × 224) using linear interpolation, preserving spatial-temporal patterns.

Self-Supervised Skeleton Representation Learning

This approach leverages powerful vision-pretrained models like MAE and DiffMAE for skeleton representation learning, effectively transferring rich visual-domain knowledge. The core idea is to train these models using self-supervised objectives without manual annotations, bridging the modality gap between images and skeleton sequences.

Masked Autoencoders (MAE) reconstruct masked image patches from visible context. By converting skeleton sequences to S2I images, MAE can be initialized with ImageNet-pretrained weights and then perform skeleton pretraining via masked reconstruction loss. DiffMAE enhances this framework by incorporating iterative denoising processes inspired by diffusion models, reconstructing masked regions through a denoising diffusion process conditioned on visible parts. Both models learn transferable representations that are highly effective for downstream skeleton tasks.

Optimizing Mask Sampling Strategies

The effectiveness of masked modeling depends heavily on the masking strategy employed during pretraining. For skeleton data, various strategies were investigated, including:

• Random Masking: Standard approach, randomly masking image patches without considering spatial relationships.
• Block Masking: Masks contiguous regions to increase task difficulty and encourage learning of stronger local structural relationships.
• Joint Masking: Skeleton-specific, focusing on the spatial domain by randomly masking joints, challenging the model to infer missing joint positions based on articulated structure.
• Temporal Masking: Targets the temporal dimension by masking entire frames or temporal slices, encouraging capture of dynamic motion patterns from partial sequences.

Findings: Extensive ablation studies showed that Random Masking with a 75% ratio consistently outperformed other strategies, indicating its robustness in capturing diverse patterns within the S2I representation.

Universal Representation & Cross-Format Learning

A key advantage of S2I is its ability to support universal skeleton representation learning. Unlike conventional methods that are tied to homogeneous skeleton formats and fixed joint numbers, S2I provides a format-agnostic solution.

This means S2I can process skeleton data with varying joint configurations (e.g., 25-joint, 20-joint, 13-joint) without needing dataset-specific joint definitions or architectural modifications. By abstracting skeleton data into a consistent image-like structure, S2I enables:

• Seamless Cross-Format Transfer Learning: Models trained on one skeleton format can effectively generalize to others, overcoming limitations of existing methods that often lose information during joint downsampling or interpolation.
• Joint Pretraining Across Diverse Datasets: Aggregating training data from multiple heterogeneous skeleton datasets (e.g., NTU120, PKU-I, PKU-II, Toyota, NW-UCLA) significantly boosts performance and enhances model generalization and robustness.