Vision-Language Models (VLMs)

DEX-AR was rigorously evaluated on ImageNet, VQAv2, and PascalVOC. It consistently outperformed baselines across perturbation-based metrics (using a novel normalized perplexity measure) and segmentation-based metrics. This robust validation confirms DEX-AR's effectiveness in providing accurate and relevant explanations for diverse VLM architectures and tasks. The improvements demonstrate its superior ability to identify truly influential image regions for VLM outputs, enhancing trust and reliability.

Key design choices in DEX-AR, such as dynamic head filtering and sequence-level token filtering, were validated through extensive ablation studies. The dynamic head filtering mechanism identifies attention heads focused on visual information, significantly improving Signal-to-Noise Ratio from 1.64 to 3.64 for LLaVA. The sequence-level filtering effectively distinguishes visually-grounded tokens from purely linguistic tokens, boosting SNR from 9.16 to 96.12 on PascalVOC-QA. These findings highlight the critical role of selective gradient filtering in generating accurate and meaningful attribution maps.

A qualitative analysis showcased DEX-AR's ability to localize objects with high precision, even in complex, cluttered scenes. It effectively distinguishes between visually grounded tokens (e.g., 'suit') and linguistic completions (e.g., 'case'), demonstrating robustness to occlusion and scene complexity. Failure cases also provide valuable insights into potential model biases and spurious correlations, enhancing model reliability. This comprehensive analysis underscores DEX-AR's utility as a tool for better model understanding and responsible AI deployment.