AI FOR ENGINEERING

LLM Customization Methods

The study delves into various techniques for adapting pre-trained LLMs to domain-specific tasks, contrasting approaches from training from scratch to prompt engineering, prompt learning, and fine-tuning. The aim is to identify the most effective strategy for enabling LLMs to act as specialized virtual assistants.

Each method presents a unique trade-off between performance, computational resources, and required expertise. The paper highlights fine-tuning, especially Parameter-Efficient Fine-Tuning (PEFT) like LoRA, as a robust method for achieving significant domain-specific performance gains due to its ability to incorporate extensive data without prohibitive computational costs.

Comparison of LLM Customization Methodologies

PyChrono Challenges

An analysis of the Project Chrono user forum reveals key challenges faced by users, which ChronoLLM aims to address. These challenges include difficulties in simulation setup, extensive API usage problems, and module installation/compilation issues.

The complexity and frequent changes in the PyChrono API often lead to outdated method usage and a lack of specific alignment in general LLMs, resulting in hallucinated or non-functional scripts. ChronoLLM's customization directly targets these issues by integrating up-to-date domain-specific knowledge.

Evaluation Metrics

To rigorously assess ChronoLLM's performance, a multi-faceted evaluation strategy is employed, including similarity-based, execution-based, and rubric-based LLM-as-judge approaches.

These metrics capture different aspects of code quality: similarity metrics like CodeBLEU and ROUGE-Lsum measure textual overlap; execution metrics like pass@k and compile@k ensure functional correctness; and the J-LLM (Judge-LLM) framework provides an expert-like, interpretable assessment of semantic faithfulness and best-practice adherence.

Model Performance Across Variants (Pretrain / ICL / SFT)

Practical Case Studies

The paper showcases ChronoLLM's capabilities through two practical examples: a double pendulum simulation and a MAN 10T truck model. These studies demonstrate how fine-tuned LLMs can generate functional and physically consistent PyChrono scripts from natural language prompts, even for complex systems with multiple bodies, constraints, and specialized components like tires and powertrains.

While the initial output may require minor expert review, ChronoLLM significantly reduces the manual effort in script generation, serving as a powerful assistant for engineers. The iterative refinement process through additional prompts further highlights its adaptability to specific user modifications.