From 2D Alignment to 3D Plausibility: Unifying Heterogeneous 2D Priors and Penetration-Free Diffusion for Occlusion-Robust Two-Hand Reconstruction

The Fusion Alignment Encoder (FAE) is a key innovation, integrating diverse 2D priors from vision foundation models in a compact and learnable way. It implicitly learns fused prior features from latent outputs during training, capturing consistent geometric and semantic structure without the need for heavy foundation encoders during inference.

Our penetration-free diffusion model represents a significant leap. Unlike previous diffusion methods that primarily regularize output, ours explicitly models 3D spatial interactions, transforming interpenetrated poses into physically plausible, collision-free configurations. The incorporation of collision gradient guidance during denoising further enhances its de-penetration capability, leading to robust recovery even under occlusion.

Extensive experiments on InterHand2.6M, HIC, and FreiHAND datasets demonstrate our method's state-of-the-art or leading performance in interaction alignment and penetration suppression. We achieve superior MRRPE, MPJPE, and MPVPE across various complex scenarios, including those with severe occlusions.

Qualitative results on real-world images from the internet further highlight our method's superior stability on unseen data, consistently delivering accurate and stable reconstructions even in challenging cases where existing methods exhibit misalignment, thumb distortion, penetration, and failure under occlusion. This robust performance is critical for practical applications.