Research Insights

Enterprise AI Analysis: Safety and efficacy of privacy-preserving models to create Lay summaries of brain MRI reports

Patient access to radiology reports has highlighted the need for patient-friendly communication. Automated generation of patient-centered summaries using large language models (LLMs) is a promising solution. This study evaluated the safety and effectiveness of on-premise, privacy-preserving LLMs for generating lay summaries of real French brain MRI reports for emergency presentations of headache. The study sampled 105 brain MRI reports for radiologist evaluation and a subset of 30 for non-physician evaluation. Three open-weights models (Llama 3.3 70B, Athene V2, Mistral Small) generated French lay summaries. Radiologists' mean ratings across models were high for exactness (4.10), exhaustiveness (4.34), didacticness (3.83), and readiness for clinical use (3.84). Non-physicians reported higher perceived understanding with summaries (from 2.85 to 4.27, p < 0.001) and improved correct identification rate for reports (from 75.2% to 83.6%, p < 0.001). The ability to identify causal findings also improved (from 80.6% to 84.8%, p < 0.001). The overall error rate in LLM-generated lay summaries was 19.7% (62/315), warranting expert oversight. The study concludes that privacy-preserving LLMs can improve perceived and objective understanding of brain MRI reports for highly educated non-physicians, but medical errors and linguistic issues necessitate a 'human-in-the-loop' framework for clinical safety.