Optimizing Competitive Programming Prompts

The ALE Agent, tackling competitive programming on AtCoder Heuristic Contest, achieved a 13.6% improvement in acceptance rate through prompt optimization. ARTEMIS transformed vague instructions like "consider edge cases" into structured decomposition steps and systematic validation strategies.

This led to more robust and correct algorithmic implementations. While requiring substantial computational resources (411.2 hours), the practical improvements in a competitive domain justify the investment, demonstrating the value of evolutionary prompt engineering for complex reasoning tasks.

Example Prompt Evolution:

Before: "Generate a solution for the given problem. Consider edge cases and optimize for performance. Implement the algorithm efficiently."

After: "Decompose the problem into sub-components: (1) identify input/output patterns, (2) detect algorithmic category (graph, DP, greedy), (3) enumerate edge cases explicitly (n=0, n=1, maximum bounds), (4) implement with clear variable naming and modular functions. Validate against sample inputs before submission."

Systematic Performance Optimization with Mini-SWE

The Mini-SWE Agent demonstrated a statistically significant 10.1% performance improvement in code optimization tasks on the SWE-Perf benchmark. ARTEMIS transformed generic "general improvements" strategies into targeted, bottleneck-driven optimization approaches.

This included systematic complexity analysis before optimization, data structure selection based on access patterns, and domain-specific techniques like vectorization and caching. Project-level results showed significant gains, for instance, a +62% relative improvement for `astropy`. The `astropy` array comparison example saw 6 key improvements, including early identity checks, optimized array dtypes, batch processing, and vectorized comparisons, leading to substantial performance gains.

Balancing Accuracy and Cost for Mathematical Agents

For the CrewAI Agent, ARTEMIS achieved a dramatic 36.9% reduction in token cost for mathematical reasoning tasks, with a statistically insignificant decrease in accuracy. This showcases ARTEMIS's capability for multi-objective optimization, prioritizing cost efficiency when baseline performance is already robust.

The optimization involved prompt refinement and token limit adjustments, leading to more efficient execution of medium-difficulty problems and, notably, the intentional failure of exceptionally expensive (and likely incorrect) problems at zero cost. This trade-off reflects a strategic optimization aligned with business objectives to reduce operational expenses.

Optimizing Primary-Level Math Solving with Smaller Models

The MathTales-Teacher Agent, powered by a smaller open-source model (Qwen2.5-7B), achieved a significant 22% accuracy improvement and a 12.1% increase in completion rate on GSM8K primary-level mathematics problems. ARTEMIS successfully enhanced its simplistic prompts with explicit verification steps and decomposition strategies.

This approach effectively addressed challenges like agents getting stuck in execution loops or producing confident but incorrect numerical calculations, demonstrating ARTEMIS's ability to optimize agents based on smaller, local models, thereby enhancing performance without reliance on commercial APIs and their associated costs.

Current Limitations and Future Work

While ARTEMIS proves highly effective, limitations include varying optimization effectiveness based on initial configuration quality, potential generalization issues (optimizations may be dataset-specific), and substantial computational costs for some complex benchmarks.

Future work will focus on three key directions:

• Planning Agent Integration: Leveraging ARTEMIS's planning agent with genetic algorithms and Bayesian optimization for complex data science tasks.
• Predictive Metrics: Developing metrics to assess optimization potential upfront, estimating ROI through prompt specificity and configuration entropy analysis.
• Transfer Learning: Investigating few-shot optimization across related agent domains to reduce evaluation costs significantly while maintaining quality.