Enterprise AI Analysis

Multimodal Sentiment Analysis (MSA) aims to recognize emotions by leveraging language, visual, and acoustic signals. Early methods relied on simple feature-level fusion, which struggles to capture complex cross-modal dependencies. Subsequent works introduced tensor-based models and attention mechanisms to better model intra- and inter-modal interactions. MSA remains challenging due to modality-specific noise, temporal misalignment, and semantic inconsistencies across modalities.

Disentangled Representation Learning approaches adopt disentangled representations that separate shared and modality-specific components. MISA, FDMER, and DLF are examples. However, these methods often impose a binary partition (common vs. private) and overlook partially shared signals. Our TSD framework addresses this limitation by explicitly introducing pairwise shared subspaces.

Fusion Strategies: Beyond early fusion, attention-based fusion, modality-specific transformations, and decision-level fusion with gating have been proposed. Most methods assume modality contributions are either fully shared or independent, implicitly collapsing partially shared signals into private spaces or enforcing global sharing. In contrast, our TSD framework explicitly introduces pairwise shared subspaces and couples them with the SACA fusion module for fine-grained modeling.

Our TSD framework explicitly disentangles multimodal representations into three complementary subspaces: (i) a common subspace capturing global semantic cues shared across all modalities, (ii) submodally shared subspaces that model interactions present only in subsets of modalities, and (iii) modality-specific (private) subspaces that preserve unique and discriminative information for each modality.

We first encode linguistic, visual, and acoustic inputs into modality-specific high-level representations and project them into a unified feature space. Dedicated decoupling modules then map these representations into the three subspaces. The common subspace aggregates modality-invariant information. Submodally shared subspaces capture associations present in two out of three modalities. Private subspaces retain complementary features unique to each modality. Finally, we fuse the outputs from all three subspaces and map the fused representation to either a categorical label or a continuous sentiment score.

Experiments on CMU-MOSI, CMU-MOSEI, and MIntRec demonstrate that TSD consistently achieves state-of-the-art performance across all key metrics, outperforming strong baselines like EMOE. TSD shows strong robustness to temporal perturbations under both aligned and unaligned settings, with consistent gains in Acc-7 and F1-score, indicating better fine-grained sentiment modeling.

Ablation studies confirm the importance of all modalities and representation spaces (common, private, sub-shared). Removing any of these leads to noticeable performance drops, validating the explicit modeling of multi-granular cross-modal sentiment cues. The SACA fusion mechanism consistently outperforms other baselines like Sum, Concat, and CMAF.

Qualitative analysis shows TSD better captures cross-modal cues like sarcasm, producing sentiment predictions closer to ground truth. t-SNE visualizations show the full TSD model yields a more compact and continuous sentiment gradient. Sensitivity analysis confirms robustness to hyperparameter selection.