Research on Multimodal Gait Recognition Based on Silhouette and Skeleton

GaitMHA: Optimizing Multimodal Fusion for Robust Identity Verification

GaitMHA's core innovation lies in its sophisticated approach to integrating diverse biometric modalities. By treating silhouette, bone, and joint data as distinct yet interconnected information streams, the model leverages specialized processing for each before intelligently fusing them. This contrasts with earlier methods that often performed generalized fusion, leading to information loss or suboptimal weighting of crucial features. The Block-wise Self-Attention mechanism is key here, allowing the model to dynamically prioritize and combine features based on their relevance to the recognition task, enhancing both accuracy and adaptability to varying conditions like clothing changes or carrying items. The G-Module further refines this by providing a stable learning environment, preventing the pitfalls of complex neural architectures.

• Skeleton Decoupling for Granular Feature Extraction

  Unlike traditional methods that treat skeleton data as a monolithic entity, GaitMHA decouples it into bone and joint components. This allows for specialized feature extraction tailored to the unique kinematic properties of each, significantly enhancing the granularity and relevance of the extracted information. Joints provide precise positional data, while bones offer structural and proportional insights, together forming a more comprehensive skeletal representation.

• Block-wise Self-Attention for Adaptive Modality Fusion

  The introduction of a Block-wise Self-Attention module is central to GaitMHA's superior performance. Instead of fixed fusion weights, this mechanism adaptively allocates importance across silhouette, bone, and joint features. It operates by partitioning feature vectors into segments, each processed by a dedicated self-attention unit with customizable parameters. This enables the model to identify and prioritize the most discriminative features from each modality, optimizing their integration.

• Gating Module (G-Module) for Enhanced Model Robustness

  To counteract common issues in deep learning like overfitting and gradient explosion, GaitMHA incorporates a Gating Module (G-Module). This module leverages residual network principles, forcing the network to focus on discrepancies between input and output, rather than merely reconstructing inputs. The skip connections facilitate gradient flow, ensuring stable training and robust performance, particularly under challenging conditions like viewpoint occlusion or covariate changes.