Enterprise AI Analysis

The core of this research is a domain-adaptive pretraining strategy that optimizes the vision encoder for fine-grained understanding of anatomical structures. It adapts a multi-task generative framework, leveraging region-level supervision. The model, composed of a Siglip vision encoder and a transformer decoder, processes chest X-ray images to produce visual tokens. These tokens serve as cross-attention input for the decoder, which performs various location-aware tasks:

The proposed location-aware pretraining framework achieves state-of-the-art performance on the MIMIC-Diff-VQA dataset, significantly outperforming baselines across multiple natural language generation metrics. Key improvements include a 24.4% relative gain in CIDEr and a 7.8% relative gain in BLEU-4 compared to the strongest existing baselines. This demonstrates the model's superior ability to generate accurate and contextually relevant answers when reasoning about differences in medical images.

Qualitative analysis demonstrates the model's enhanced ability to identify and attribute clinically relevant changes between reference and main images. In Case I, both our model and ReAl correctly identified 'lung opacity and atelectasis'. In Case II, while ReAl missed 'pleural effusion', our model correctly identified 'atelectasis'. Case III, a more challenging scenario, showed that both models struggled with some findings, but our model accurately reported 'atelectasis and pleural effusion' and correctly identified 'lung opacity and pneumonia' as missing, while ReAl's output was less precise. These examples highlight the robustness of location-aware pretraining in accurately capturing subtle pathological differences.

An ablation study on pretraining tasks confirmed the importance of each location-aware objective. Removing all location-aware tasks led to a substantial performance drop. Specifically, excluding Automatic Referring Expressions (AREF) resulted in the largest degradation, highlighting its critical role in region-level reasoning. The optimal performance was achieved when all location-aware tasks (AREF, GCAP, CAREF, and Captioning) were synergistically combined. Furthermore, using a higher image resolution (448x448) consistently outperformed 224x224, and a moderate masking ratio (25%) for parallel prediction yielded the best downstream results, indicating an effective balance between reconstruction and information preservation.