Enterprise AI Analysis

LLM Performance Comparison

LLaMA2 generally outperformed other LLMs on general device features, while BERT and RoBERTa showed better performance for clinical features like 'Users' and 'Health conditions', indicating their strength in summarization. All models struggled with nuanced ML-specific details, highlighting the need for domain-specific optimization.

Insights and Limitations of LLM Application

Our findings showed that decoder models like LLaMA2, though requiring significant computational resources, outperformed others in extracting device characteristics from approval documents, including long sections, due to their larger 4,096-token limit. However, LLaMA2 struggles with precise information representation, often introducing irrelevant content and producing overly creative response, leading to low precision in BLEU scores and limiting its clarify and feasibility in extracting ML and clinical features. Encoder-based LLMs like BERT produce concise outputs but constrained by small architecture, hindering performance on tasks requiring longer text processing, such as device descriptions. However, BERT performed relatively well at summarizing shorter text spans, such as health conditions and users [10].

Overall, all the LLMs, primarily trained on general corpora like Wikipedia, struggle to address domain-specific features critical for analysing ML applications in clinical settings [9]. In addition, the inconsistent structure of FDA documents, with irregular sections, non-standard layouts, and inclusion of figures and tables, poses challenges for designing effective prompt templates, limiting LLM performance. Further, the exclusive use of objective performance metrics, focused on word overlapping rather than semantic meaning, restricts evaluation. Subjective measures would provide better assessment [16].

Future Directions and Recommendations

This study assessed the feasibility of leveraging LLMs to automate the extraction of critical information from FDA-approved ML medical device documents. Decoder-based LLMs showed strong performance in retrieving explicit information such as device characteristics and ML functions, though at the cost of high computational demands. In contrast, encoder-based models (e.g., BERT and RoBERTa) were more efficient and better at inferring clinical context, including intended users and applications. However, all models struggled with extracting nuanced ML-specific details, highlighting the need for domain-specific optimization strategies such as finetuning. These findings establish a benchmark for applying LLMs in regulatory document analysis and suggest a promising role for tailored LLM pipelines in enhancing the transparency and oversight of ML-enabled medical devices.

This study highlights the importance of clear instructions and integrating domain knowledge to improve the performance of generative LLMs. Future work could explore approaches such as Retrieval-Augmented Generation, which allows the model to access precise and authoritative knowledge sources by integrating domain-specific data (e.g., annotated FDA approvals, device manuals, regulatory guidelines) into the response generation process [18]. Alternatively, fine-tuning LLMs with instructions or labels can help the model better understand terminology, context, and relationships, bridging the gap between general training and domain-specific tasks to improve accuracy and reliability in extracting specific device features. However, the benefits must be weighed against the exponential computational, environmental and privacy concerns [19].