Enterprise AI Analysis

Automated Environment Generation Pipeline

Our methodology focuses on a scalable pipeline for synthetic ML task generation, designed for diversity and validity. It consists of three phases:

1. Environment Synthesis: Automatically samples diverse ML topics, proposes real HuggingFace datasets, and generates task-specific configurations and starter code, including baseline implementations and evaluation files.
2. Environment Verification: Each newly generated task is plugged into MLGym and run with a GPT-5 agent to obtain baseline performance and at least one trajectory. Errors are fed back into a self-debugging loop for correction, ensuring task validity without human intervention.
3. Trajectory Generation & Filtering: Synthetic tasks are run in parallel on an HPC cluster to collect a large number of agent trajectories (e.g., 256 per task). These are then filtered for successful submissions and length, forming a high-quality SFT training dataset.

Validated Performance on MLGym Benchmark

We applied our method to the MLGym benchmark, which comprises 13 diverse ML tasks. Our environment synthesis system produced approximately 500 tasks, generating about 34,000 agent trajectories from a GPT-5 teacher model.

These trajectories were used to fine-tune student models, Qwen3-4B and Qwen3-8B. The results demonstrate significant performance gains on MLGym, with the AUP (Area Under Performance Curve) metric increasing by 9% for Qwen3-4B and 12% for Qwen3-8B. Our trained models outperformed baselines on 9 out of 13 individual tasks.

Strategic Implications and Future Directions

This work provides a practical path towards building truly autonomous AI scientists. While the current pipeline shows strong results, we acknowledge limitations such as reliance on a single benchmark and potential biases inherited from the teacher model. Future work will explore more complex codebases and integrate reinforcement learning for genuine discovery.

The ability to scale task synthesis and generate agentic experience for a wide variety of ML tasks positions this approach for broader applications across various benchmarks and for optimizing the discovery of novel ideas beyond current solutions.

Synthetic Task Scaling for Advanced AI Agents

We presented a scalable pipeline for training machine learning research agents via synthetic task scaling. This approach automatically generates diverse ML tasks compatible with the SWE-agent framework by sampling topics, proposing and validating real HuggingFace datasets, and synthesizing full runnable environments including configs, starter code, and evaluation scripts.

By generating roughly 500 synthetic ML tasks and collecting ~30k–34k teacher trajectories from GPT-5, fine-tuning Qwen3-4B and Qwen3-8B on these trajectories leads to consistent gains on the MLGym benchmark, improving aggregate AUP by 9% and 12% respectively, and improving performance on the majority of individual tasks. These results suggest that synthetic environments can provide effective training signal for long-horizon agent behaviors such as iterative debugging, experimentation, and implementation refinement.

Contextualizing AI in Scientific Discovery

Our work builds upon recent advancements in LLM-based agents for scientific research. We differentiate from existing systems that primarily focus on ideation or final outputs by providing an end-to-end pipeline that enables agents to learn from the full iterative research cycle.

We draw insights from benchmarks like MLE-Bench and PaperBench, which evaluate agents' ability to reproduce ML engineering and research workflows. By generating diverse, grounded tasks, our approach offers a novel way to address the challenge of scaling agentic experience beyond static corpora.