Research Paper Analysis

LMMs: Bridging Modalities for Deeper Understanding

Large Multimodal Models (LMMs) extend LLMs by integrating various data types such as text, images, audio, and video. They enable a more comprehensive understanding and generation of diverse content by creating a unified multimodal embedding space.

• Multimodal Capabilities: Processing visual inputs, generating textual descriptions from images, and handling audio data.
• Unified Embedding Space: Separate encoders for each modality generate data-specific representations, which are then aligned into a cohesive multimodal space, integrating and correlating information seamlessly.
• Examples: Notable LMMs include Gemini, GPT-4V, ImageBind, BLIP-2, LLava, and Qwen-VL, showcasing zero-shot or few-shot context learning abilities.

Multimodal Sentiment Analysis: Core Challenges & Fusion

Multimodal Sentiment Analysis (MSA) combines multiple modalities (e.g., image, text, audio) to enhance sentiment classification accuracy, reflecting human emotion processing more closely. Unlike text-only scenarios, MSA presents unique challenges:

• Complexity of Sentiment Semantic Representation: Sentiment semantics derived from various modality representations are complex to select and fuse.
• Complementarity of Sentiment Elements: Other modalities can provide effective supplements to often shorter, less informative textual expressions.
• Inconsistency in Sentiment Expression: Conflicts in sentiment expressions among modalities (e.g., irony) require careful disambiguation.

The core of MSA involves independent representation of single-modal semantics and their fusion. Fusion types include Feature Layer Fusion (Early Fusion), Algorithm Layer Fusion (Model-level Fusion), and Decision Layer Fusion (Late Fusion).

Image-Text Sentiment Analysis: From Coarse to Fine-Grained

This category focuses on sentiment analysis from image-text pairs, primarily in social media and e-commerce contexts. It is divided into coarse-grained and fine-grained levels.

Coarse-grained Level:

Involves sentiment classification (positive, neutral, negative) and emotion classification (happy, sad, angry, etc.) at a sentence level. Early models were feature-based, evolving to neural network models using attention mechanisms. LLMs enhance this by leveraging contextual world knowledge (e.g., WisdoM) and prompt-based fine-tuning (e.g., UP-MPF, PVLM). Datasets include MVSA, TumEmo, MEMOTION 2, MSED.

Fine-grained Level:

Focuses on analyzing finer sentiment elements like aspect term (a) and sentiment polarity (p). Subtasks include:

• Multimodal Aspect Term Extraction (MATE): Extracting aspect terms from a sentence given an image.
• Multimodal Aspect-based Sentiment Classification (MASC): Identifying sentiment polarity for a given aspect term.
• Joint Multimodal Aspect-Sentiment Analysis (JMASA): Simultaneously extracting aspects and their polarities.

LLMs (e.g., A2II, GMP) are being adapted to these tasks to mitigate limitations of modality fusion and enhance performance.

Audio-Image-Text Sentiment Analysis (Video): Temporal Dynamics

Video-based sentiment analysis differs from image-text analysis by emphasizing facial expressions and body movements in video sequences, and considering temporal intra-modal emotional factors and alignment over time. Tasks include sentiment classification (3, 5, or 7 categories) and emotion classification (multi-label or single-label).

Methods focus on two core themes:

• Cross-modal Sentiment Semantic Alignment: Explores associations between emotional information across modalities, analyzes alignment relationships, and reduces semantic distance. Approaches include attention-based alignment (e.g., MulT), contrastive learning-based alignment (e.g., CLIP-inspired methods), and cross-domain transfer learning-based alignment.
• Multimodal Sentiment Semantic Fusion: Aggregates sentiment information to achieve comprehensive understanding. Methods include tensor-based fusion (e.g., LMF), fine-grained temporal interaction modeling fusion (e.g., RAVEN), and pre-trained model-based fusion (injecting multimodal information into language models).

Datasets include ICT-MMMO, IEMOCAP, CMU-MOSI/MOSEI, MELD, CH-SIMS, M3ED, MER2023/2024, EMER, CMU-MOSEAS, UR-FUNNY.

Multimodal Sarcasm Detection: Unveiling Irony

Multimodal sarcasm detection aims to identify sarcastic meaning in text associated with an image, crucial for disambiguating inconsistent sentiment signals across modalities. Sarcasm can involve either complete sentiment conflict or instances where some modalities convey 'neutral' sentiment while others are positive/negative (implicit sentiment expression).

Key methodologies include:

• Multi-task frameworks: Simultaneously recognizing sarcasm and classifying sentiment polarity.
• Incongruity detection: Capturing emotional semantic cues across modalities.
• Attention mechanisms: BERT-based models with cross-modal and text-oriented co-attention.
• Graph Neural Networks (GNNs): Building heterogeneous intramodal and cross-modal graphs to learn inconsistency relationships.
• Prior knowledge integration: Utilizing knowledge bases like ConceptNet to determine image-text relevance.

LLMs (e.g., CofiPara) are leveraged using prompts to cultivate divergent thinking and elicit relevant knowledge for judging irony, viewing LMMs as modal converters to textualize visual information for cross-modal alignment. Datasets include MMSD, MMSD2.0, MUSTARD.