Enterprise AI Analysis

The development of robust Question Answering (QA) systems is critical for human-level AI. While Large Language Models (LLMs) have advanced, specialized domains like medical QA and low-resource languages like Romanian pose unique challenges. Medical QA requires deep understanding of clinical terminology and reasoning, with high accuracy being paramount due to serious consequences of mistakes. Existing multilingual QA datasets largely omit Romanian. MedQARo addresses this gap by providing the first large-scale medical QA benchmark in Romanian, grounded in oncology case summaries.

MedQARo data was manually extracted and annotated by seven physicians from clinical documents (epicrises) from two medical centers in Bucharest, Romania. The dataset includes 1,242 oncology patients (796 breast, 215 lung, 231 other cancers) resulting in 105,880 high-quality QA pairs. Data splits are patient-level to prevent leakage, including in-domain and cross-domain test collections for generalization assessment. Four open-source LLMs (RoLLaMA2-7B, RoMistral-7B, Phi-4-mini-instruct, LLaMA3-OpenBioLLM-8B) were fine-tuned using LoRA and evaluated in zero-shot settings. GPT-5.2 and Gemini 3 Flash were also evaluated zero-shot via APIs. Evaluation metrics include F1 score, Exact Match (EM), BLEU, and METEOR. Ethical approval was obtained, and data is anonymized and publicly available.

Fine-tuned models significantly outperform zero-shot models and baselines on MedQARo. The Q+E+A prompt format yields higher performance than E+Q+A. RoMistral-7B (fine-tuned) achieves the best performance with an F1 score of 0.671 on the in-domain test set, demonstrating robustness. Phi-4-mini-instruct performs better when focusing on the first 2,048 tokens of an epicrisis, suggesting context trimming as a regularization technique. Cross-domain evaluation reveals a noticeable performance drop for all models, highlighting challenges in generalization. API-based LLMs (GPT-5.2, Gemini 3 Flash) perform worse than fine-tuned open-source LLMs, underscoring the importance of task-specific and domain-specific fine-tuning.

MedQARo is a challenging benchmark, with even the best fine-tuned model achieving an F1 of only 0.671 in-domain and 0.445 cross-domain. This gap indicates ample room for improvement. Key findings include: 1) Long prompts aren't always better; optimal context length is crucial. 2) Language-specific LLMs (RoMistral-7B, Phi-4-mini) are better starting points than medical domain-specific (OpenBioLLM-8B) for Romanian QA. 3) Fine-tuning is critical, outperforming zero-shot significantly. Future work will explore Retrieval-Augmented Generation (RAG) to improve generalization and handle extensive clinical narratives more effectively without context trimming issues.