Enterprise AI Analysis

In traditional software engineering, behavior-preserving system transformation is a well-understood concept. As first introduced by Opdyke (1992), refactoring in object-oriented programming involves systematically restructuring code without altering its external behavior. However, a learning-enabled software systems (LESS)—where Machine Learning (ML) models and data drive system behavior—refactoring is far more ambiguous. How can we verify that fine-tuning produces a trustworthy transformation that maintains both the system's behavioral integrity and interpretable decision-making? The uncertainty is especially problematic in high-autonomy domains such as safety-critical infrastructure, finance, and healthcare, where ML components must remain reliable and explainable under continuous evolution. Yet, despite this need, there is currently no empirical baseline for what counts as a safe or semantics preserving change when updating models, training data, or documentation. This gap raises risks not only in performance regression but also in trust, reproducibility, and downstream system reliability.

To mitigate these risks, it is increasingly critical to understand how ML models evolve while retaining their original intent. Unlike traditional software artifacts, ML models are not static—they are rapidly updated through fine-tuning, performance optimization, and documentation updates. Our goal in this paper is to uncover patterns and boundaries of semantic preservation during system transformation in widely used pretrained ML models that are hosted on Hugging Face. The Hugging Face platform offers a uniquely rich and open environment to observe these dynamics at scale, allowing researchers to trace intra-repository evolution through version-controlled Model Cards and commit histories.

This study addresses its research questions by detecting semantic drift across versions of ML/DL models using publicly available metadata and documentation from Hugging Face. For RQ1, millions of pre-trained model repositories were curated and filtered to retain only non-trivial cases. For RQ2, functional behavior metrics (accuracy, F1-score) were extracted from versioned Model Card README documentation, and metric trends analyzed across commits to evaluate semantic preservation. The methodology comprises metadata collection and filtering, performance metric extraction, and evaluation case studies. The pipeline aims to quantify practical refactoring thresholds in LESS.

The process begins by accessing the HF hub via the HfApi interface, retrieving model and user metadata from over 1.7 million models. A multi-stage filtering strategy was applied: 1) excluding repositories with missing or empty Model Card files, 2) discarding models lacking valid architecture fields in config.json (reducing the dataset to 751,964 models), and 3) filtering non-functional keywords in versioned commits. This process resulted in a subset of 536 models with 4297 metrics for detailed analysis.

Performance metrics like accuracy, precision, F1-score, and loss were extracted. Initially, a rule-based extraction pipeline was used, but due to the diversity of documentation formats, a GPT-5-involved API call was adopted for more robust and accurate extraction, generating structured summaries of performance metrics and overcoming static regex limitations. Semantic preservation is defined as consistent predictive performance across model versions.

The empirical study identifies several patterns of refactoring maintenance in Hugging Face model repositories. Automated metric extraction successfully identifies both positive drift (optimization) and semantic preservation patterns. Commit histories and public documentation serve as valuable, though underexplored, artifacts for refactoring analysis in ML, revealing common refactoring patterns like 'update' (95.6% of commits) overshadowing explicit mentions of 'refactor' or 'optimize' (each 0.1%). This highlights that refactoring behaviors are often under-documented or embedded within general-purpose commit labels.

Documentation quality significantly impacts the observability of model evolution, underscoring the need for community-wide standards. The framework enabled detection of different semantic drift patterns: Optimized Drift (metrics improve steadily), Semantic Preservation (no significant change despite updates), and Performance Degradation (performance declines despite refactoring efforts). Overall, 16.6% of the analyzed models exhibited semantic drift, though statistical significance was limited due to data variability. The pipeline provides concrete evidence of semantic preservation in LESS by showing where key performance metrics remain stable across model updates.

Analysis of commit messages across sampled repositories revealed prevalent refactoring patterns. The vast majority of commits (95.6%) utilize the generic keyword 'update'. Other common, yet less frequent, keywords include 'test' (2.0%), 'style' (1.3%), and 'improve' (0.8%). Explicit mentions of non-functional maintenance activities such as 'refactor', 'optimize', or 'security' are notably rare, each accounting for only 0.1% of commits. This imbalance indicates that refactoring behaviors are often under-documented or embedded within general-purpose labels, making it challenging to precisely trace the intent and impact of changes. This highlights a critical need for more sophisticated information distillation tools to confirm whether changes preserve or improve semantic functionality.

By tracking performance metrics over time, three recurring patterns of semantic drift were observed across models:

• Optimized Drift: Metrics steadily improved across commits, often reflecting intentional tuning. For example, some models showed a 40.09% accuracy delta in image detection due to fine-tuning.
• Semantic Preservation: No significant change in performance despite updates. This indicates successful maintenance of the model's original intent. For instance, several models maintained high accuracy (e.g., 98% accuracy for image detection and 79.6% for text classification) over multiple commits.
• Performance Degradation: Performance did not always improve, sometimes fluctuating or declining. For example, a tabular data classification task showed accuracy fluctuating from 85.5% down to 84.8% over multiple commits.

Overall, 16.6% of the analyzed models reported at least one instance of semantic drift. However, confidence intervals for primary outcomes often included zero, suggesting that observed changes were not always statistically significant, likely due to small sample sizes or high data variability across diverse models and tasks.

The methodology's reliance on documentation quality and commit accessibility on Hugging Face presents limitations. Inconsistent use of structured evaluation metadata, varying README.md file formats, and diverse metric formats introduce noise despite extraction efforts. While aggregate metrics offer a starting point, they can mask bias or insufficiency at a system level.

Future work will address these by expanding beyond aggregate metrics to conduct more granular analyses, examining model predictions on specific data slices to detect subtle shifts in subgroup performance, and including dimensions such as decision boundary shifts and subgroup performance changes. The framework will also be extended to explore fairness, efficiency, and other multi-objective dimensions, aiming to improve robustness across diverse documentation styles.