AI/ML Research Analysis

MM-CoT: A Discriminative Verification Task

Contemporary Multimodal Models (MMs) exhibit remarkable fluency in generating Chain-of-Thought (CoT) rationales for visual tasks. However, this surface-level sophistication often conceals deeper weaknesses in reasoning integrity—particularly the lack of genuine visual grounding and logical coherence. Models frequently produce plausible narratives that reference nonexistent visual elements or violate causal principles, exposing a disconnect between fluent description and faithful inference.

To address these issues, we introduce MM-CoT, a diagnostic benchmark that redefines multimodal evaluation from open-ended generation to verification-based reasoning. Instead of rewarding fluent explanations, MM-CoT asks whether a model can verify a reasoning chain for both visual fidelity and logical soundness, thereby revealing reasoning gaps overlooked by prior benchmarks (see Sec. 2).

Benchmark Formalism and Construction

At the core of MM-CoT is a formalization of what constitutes a valid reasoning process. We define a reasoning chain as a Logical-Sequential Chain c = (E1 → E2 → → En). For diagnostic clarity, MM-CoT adopts a triadic form c = (A → B → C'), where A represents an initiating condition, B a mediating event, and C an outcome. Given a visual input V, a chain is valid if it satisfies two orthogonal constraints:

• Visual Grounding Constraint (vis): every event must be factually anchored to observable evidence in V. Formally, vis (C, V) = 1 iff all Ei are visually supported without hallucination.
• Logical Coherence Constraint (Φlog): the causal or temporal transitions between events must be plausible. Plog(c) = 1 iff all transitions E¿ → Ei+1 adhere to physical and commonsense principles.

A chain c* is uniquely valid when V(c* | V) = Фvis (c*, V) ^ Φlog(c*) = 1. (1) The model's task is to identify c* from a set of candidate chains, each distractor violating exactly one constraint. This design isolates two distinct reasoning demands—visual fidelity and logical progression—and enables targeted diagnosis of where the reasoning process fails.

Benchmark Construction Pipeline

To operationalize this formalism, we construct a dual-modality benchmark that evaluates reasoning over both static and dynamic visual contexts. Each instance contains a single valid chain c* and several adversarial distractors that violate either vis or ¬log. The overall generation pipeline is summarized in Algorithm 1 and qualitatively illustrated in Figs. 2 and 3. Distractors are written to be linguistically similar to the valid chain yet fail in targeted ways, preventing models from relying on superficial textual heuristics and compelling them to perform fine-grained visual and causal reasoning.

3.1.1. Image-based reasoning chains.

The image subset comprises 5,615 instances derived from Flickr30k [36], evaluating reasoning over latent dynamics in static imagery. The key challenge is to infer physically and socially coherent event sequences from a single frame, where many causal relations are implicit rather than explicitly observed. For each instance, distractors target two failure modes: hallucination of nonexistent or contradicted elements («Фvis) and implausible relational reasoning (log). The complete data flow for image-based MM-CoT construction is depicted in Fig. 2, corresponding to Algorithm 1.

3.1.2. Video-based reasoning chains.

The video subset comprises a large-scale collection of instances from ShareGPT4Video [5], assessing reasoning over explicit temporal evolution. Models must track entity states across frames and infer causal progressions over time. Distractors introduce temporal or causal perturbations, such as swapped event order, removed mediating steps, or counterfactual interventions that break the original dynamics.

Experiments and Main Results

4.1. Experimental Setup

To rigorously assess the effectiveness of MM-CoT in discriminating visual reasoning capabilities, we conduct a comprehensive evaluation of contemporary vision-language systems. On the proprietary side, we benchmarked GPT-5 [1], Gemini-2.5-Pro [41], Claude-Sonnet-4, and Grok-2-Vision-1212, all invoked via a unified OpenRouter interface. On the open-source side, we locally deployed models on A100 GPUs, including Qwen2.5-VL-72B [3], LLAMA-3.2-90B [43], GLM-4.5V [40], InternVL3-8B / InternVL3.5-8B [59], Ovis-2.5 [32], LLaVA-1.5-7B [27], and Idefics2-8B [18]. To ensure fair comparison, we strictly aligned inference configurations across all systems—using identical prompt templates, decoding temperatures, maximum generation lengths, and visual input specifications—thereby minimizing extraneous variance attributable to implementation or parameter differences.

4.2. Main Result

The results in Table 1 demonstrate that MM-CoT effectively differentiates model capabilities across both image-based and video-based reasoning tasks. In image reasoning, models exhibit relatively comparable performance in object recognition and semantic grounding; however, the performance gap becomes more pronounced when shifting from single-step answering (Single) to multi-step chain-of-thought reasoning (Multi). This indicates that MM-CoT reveals differences in semantic consistency and reasoning-chain stability, rather than merely measuring surface-level recognition ability.

In video reasoning, these differences are further amplified. As the temporal span increases from short to long videos, most models show substantial degradation under Medium/Hard conditions, while Human performance remains consistently strong. This highlights a persistent limitation of current LVLMs in cross-frame state tracking and causal event reconstruction. Notably, certain models perform better on long videos than on short ones. This does not imply superior long-range temporal reasoning; rather, longer videos often contain stronger narrative priors, enabling models to produce plausible responses through story-like completion. In contrast, short videos require fine-grained motion perception and temporal alignment, which more directly exposes deficiencies in dynamic visual reasoning. Therefore, MM-CoT effectively distinguishes narrative-style answering from genuine temporal reasoning, providing a more diagnostic and discriminative evaluation of multimodal reasoning capability.

4.3. Sensitivity Analysis of Reasoning Paradigms

To examine whether improvements in reasoning paradigms lead to quantifiable performance gains and thereby validate the sensitivity and effectiveness of MM-CoT in capturing reasoning ability, we evaluate three representative vision-language models (e.g., Qwen2.5-VL-72B, GLM-4.5V) under strictly aligned inference configurations. We compare three reasoning strategies: (i) Direct Answer, where the model produces a final prediction in a single forward pass; (ii) Chain-of-Thought Reasoning, where the model explicitly generates intermediate reasoning steps that recursively inform subsequent inference; and (iii) Reflective Reasoning, where the model performs a self-check after producing an initial answer, examining the consistency and reliability of its reasoning trace and re-deriving the answer when necessary. By conducting controlled comparisons across these paradigms under identical data and evaluation protocols, we can precisely characterize the marginal contribution of each reasoning strategy, thereby demonstrating MM-CoT's ability to sensitively capture differences in multimodal reasoning capability.

The experimental results (Table 2) reveal a consistent improvement across models when transitioning from direct prediction to more structured reasoning paradigms. For Qwen2.5-VL-72B, for instance, the Image-Multi accuracy increases from 32% under the Direct Answer setting to 44.2% with Chain-of-Thought reasoning, while performance on the Video-Hard split rises from 45.45% to 61.00%. These gains indicate that explicit reasoning steps substantially enhance the model's ability to handle complex, multi-step visual understanding. Incorporating a reflective mechanism further improves robustness in most scenarios. GPT-5, for example, exhibits a steady improvement on the Video-Extreme split, progressing from 7.54% to 10.00% and ultimately 13.82%, suggesting that self-evaluation helps correct erroneous or inconsistent initial inferences.

Importantly, this upward trend is not confined to specific models or task types; rather, it manifests consistently across architectures and difficulty levels, with the most pronounced gains observed in video-based multimodal reasoning. Overall, the monotonic performance improvements across paradigms demonstrate that enhanced reasoning structures yield quantifiable benefits. At the same time, the clear separation between paradigms confirms that MM-CoT is highly sensitive to the marginal contributions of different reasoning strategies, validating its effectiveness as a rigorous framework for evaluating multimodal reasoning capability.

4.4. Verification of Visual Dependency

To quantitatively validate the effectiveness of MM-CoT in assessing models' visual grounding capability, we introduce a Text-Only control experiment. This experiment is designed to examine whether models can still accomplish the benchmark tasks when visual signals are entirely removed. Failure under this condition indicates that MM-CoT indeed requires genuine multimodal perception and visual-semantic alignment, rather than relying on linguistic pattern recall or superficial logical inference, thereby confirming that the distractor design effectively prevents language-only shortcut reasoning. From the results in Table 3, MM-CoT effectively distinguishes models' visual reasoning capabilities across both video and image tasks. In video reasoning, models achieve substantially higher accuracy when video input is available, whereas removing video signals (Text-Only) leads to pronounced performance degradation, particularly under Medium and Hard conditions (a 30%-70% drop). This indicates that cross-frame motion tracking and temporal causal inference cannot be compensated for by linguistic priors alone, demonstrating that MM-CoT reliably evaluates dynamic visual reasoning ability. In contrast, for image reasoning tasks, models exhibit more stable performance when image inputs are present, while removing image information similarly causes a clear accuracy decline (30%-60%), confirming that this component effectively assesses static visual semantic grounding and object relationship understanding. Overall, the consistent performance collapse observed when visual modalities are removed verifies that MM-CoT successfully prevents language-only shortcut strategies and provides a faithful measurement of genuine multimodal visual reasoning ability.

MM-CoT Error Analysis

4.5. Error Analysis

To systematically uncover the core limitations of current Vision-Language Models (VLMs) in logical reasoning, we conduct an in-depth error-type analysis on two representative systems: the strongest open-source model, Qwen2.5-VL-72B, and the proprietary commercial model, GPT-5. Specifically, we randomly sample 100 failed cases from each model's incorrect predictions on our image- and video-based reasoning tasks, and annotate them according to our proposed taxonomy of causal reasoning errors. By comparing the distribution of error types across both models, we gain a finer-grained understanding of their differences in visual comprehension, causal-chain construction, counterfactual reasoning, and multimodal information integration. This analysis further reveals several structural limitations that persist in real-world reasoning scenarios, highlighting critical directions for improving future VLM architectures and training methodologies.

4.5.1. Image Error Analysis

Through a systematic examination of failure cases from Qwen2.5-VL-72B-Instruct and GPT-5 on image-based reasoning tasks, we identify five major categories of errors:

• Initial-Context Bias: The model persistently relies on the initial textual description during multi-step reasoning, failing to update its reasoning trajectory according to newly revealed visual cues.
• Visual Distraction Error: The model is misled by visually salient but causally irrelevant elements in the image, resulting in a deviation from the true causal evidence.
• Semantic Redundancy Failure: The model merely restates existing semantic content without advancing the causal reasoning chain, leading to stagnation of the reasoning process.
• Causal Inconsistency Error: Logical contradictions, incompatible conditions, or conflicts with the visual facts emerge within the reasoning chain.
• Hypothetical Condition Dependence: During conditional or counterfactual reasoning, the model over-relies on the textual premise while neglecting key visual evidence, causing the reasoning process to detach from the actual visual grounding.

From the results in Figure 4, we observe that Qwen2.5-VL-72B and GPT-5 exhibit characteristic differences and shared patterns in their error distributions for image-based reasoning tasks. First, Semantic Redundancy Failure emerges as the most dominant error type for both models (42% and 54%), indicating that current VLMs tend to repeat surface-level descriptions rather than proactively advancing the causal reasoning chain. This highlights a pervasive limitation in their ability to perform deep causal inference. Second, the proportions of Visual Distraction Error are similar across models (22% and 20%), suggesting that both systems remain vulnerable to visually salient yet causally irrelevant elements in complex scenes. The challenge of visual noise filtering thus persists as a systemic weakness shared by modern VLMs. Moreover, both models demonstrate relatively high rates of Hypothetical Condition Dependence (28% and 20%), revealing a tendency to over-rely on textual conditions while overlooking key visual evidence during conditional or counterfactual reasoning. This indicates that visual-grounded hypothesis construction is still an underdeveloped capability. In contrast, Initial-Context Bias and Causal Inconsistency Error appear at comparably low levels (2-4%), suggesting that while the models exhibit moderate stability in maintaining local coherence and logical consistency, occasional structural failures in reasoning remain unavoidable. Overall, these error patterns reveal both the capability differences and shared bottlenecks of current VLMs in visual causal modeling, text-visual information balancing, and attention filtering mechanisms. They further highlight that multi-step visual causal reasoning remains one of the most challenging and yet most essential abilities for advancing the next generation of vision-language models.

4.5.2. Video Error Analysis

Through a systematic analysis of video reasoning failures in Qwen2.5-VL-72B-Instruct and GPT-5, we observe that both models exhibit a set of causality-related errors that are inherently tied to the temporal nature of video understanding. Unlike static image reasoning, video comprehension requires the model to continuously track temporal order, action dynamics, and cross-frame causal cues—areas where current VLMs still show pronounced weaknesses. Based on our examination of failure cases, we summarize six major categories of video-specific causal reasoning errors:

• Direct Cause Omission: The model fails to identify the immediate causal antecedent that triggers the outcome, instead focusing on earlier or weakly related segments.
• Non-Causal Attribute Selection: The model incorrectly selects background elements or visually salient but non-causal attributes as causal conditions.
• Spurious Co-occurrence Attribution: The model conflates co-occurrence with causality, treating factors that merely coincide with the outcome as causal triggers.
• Causal Over-Specification: The model proposes multiple redundant or interchangeable causal conditions rather than identifying the minimal sufficient causal set.
• Inconsistent Causal Conditions: The selected causal conditions are semantically or logically incompatible, leading to internally inconsistent reasoning chains.
• Counterfactual Modeling Error: The model deviates from the true causal structure of the video when constructing counterfactual premises or generating counterfactual outcomes.

From the error distribution (Figure 5), Qwen2.5-VL-72B and GPT-5 exhibit broadly similar overall trends, while revealing notable fine-grained differences across categories. The majority of failures are concentrated in Non-Causal Attribute Selection and Direct Cause Omission. Non-Causal Attribute Selection shows the highest proportion (Qwen: 34%, GPT-5: 44%), indicating that both models frequently attend to visually salient but non-causal scene elements, failing to focus on the minimal causal set. Direct Cause Omission (20% and 22%) further shows that both models struggle to reliably identify the “immediate cause," often shifting attention to distal or accompanying factors. This highlights a structural limitation in causal chain localization. For more complex error types, the two models diverge slightly. Causal Over-Specification (18% vs. 16%) suggests insufficient causal sparsity reasoning, with models tending to produce redundant causal conditions. Inconsistent Causal Conditions (8% vs. 6%) indicates unstable causal-logic coherence, as both models occasionally generate mutually incompatible explanations. The gap in Counterfactual Modeling Error (16% vs. 8%) reveals substantial weaknesses in counterfactual premise construction and counterfactual outcome inference, with Qwen showing more pronounced deviations. Overall, both models demonstrate structural limitations in causal factor selection, causal chain localization, and counterfactual modeling. Although GPT-5 performs more robustly in several categories, both models exhibit convergent weaknesses, pointing to the need for improving causal feature selection, maintaining causal structural consistency, and strengthening counterfactual reasoning mechanisms in future VLM development.