Enterprise AI Analysis

Experimental Artifacts: Potential Confounds and Biases

These factors relate to confounders from our experimental setup or procedures that may introduce biases, or limit the external validity.

Experimentally driven overuse of AI

Some developers reported overusing AI due to the experiment, but slowdown was similar for those reporting overuse vs. normal use. The effect is unclear.

Unrepresentative task distribution

Tasks were standard but on the shorter side, excluding non-programming work. Better-scoped issues might favor AI, but also expert human performance, making the net effect unclear.

AI increasing issue scope

Developers reporting 'scope creep' with AI actually saw *less* slowdown, contradicting the idea that increased scope caused the slowdown. Mixed reports on AI's impact on scope. 47% more lines of code per forecasted hour in AI-allowed issues.

Bias from issue completion order

Developers could choose task order post-randomization. While no qualitative reports of prioritizing non-AI tasks, it cannot be fully ruled out as a bias source.

Unfamiliar development environment

Most developers used comparable IDEs (VSCode/Cursor off). Slowdown was similar (24%) for those using comparable IDEs. No clear learning effects in first 30-50 hours of Cursor usage. Unlikely to contribute.

Cheating or under-use of AI

AI used in 83.6% of allowed cases. Only 3 cheating instances (~6%) observed in AI-disallowed tasks. Unlikely to contribute to slowdown.

Issue dropout

Similar slowdown observed for developers with no accidental dropout. Intentionally dropped issues qualitatively unbiased. Unlikely to contribute.

Non-robust outcome measure

Alternative imputation methods for unreviewed issues and using screen recording time yielded similar slowdowns (14-25%). Unlikely to contribute.

Non-robust estimator

Alternative regression estimators yielded similar slowdowns. Unlikely to contribute.

Factors Enhancing Developer Performance Relative to AI

These attributes of the issues, repositories, or setting improve developer ability relative to AI, making AI less impactful.

High developer familiarity with repositories

Developers slowed down *more* on issues they were familiar with. They average 5 years experience and 1,500 commits on repositories. Expertise makes AI less helpful.

Implicit repository context

Developers report AI doesn't utilize important tacit knowledge or context, leading to less useful AI outputs. This tacit knowledge is crucial in large, mature codebases.

Limits AI Performance: Constraints on AI's Effectiveness

These attributes of the issues, repositories, or AI/environment tooling diminish AI's effectiveness relative to developers.

Large and complex repositories

Developers report AI performs worse in large and complex environments. Repositories average 10 years old with >1,100,000 lines of code. This complexity limits AI's utility.

Low AI reliability

Developers accept <44% of AI generations. Majority report making major changes to clean up AI code. 9% of time spent reviewing/cleaning AI outputs. Low reliability results in significant wasted time.

Below-average use of AI tools

Slowdown is similar for developers with prior Cursor experience, and no clear learning effect across first 30-50 hours of Cursor usage. However, one developer with >50 hours showed speedup, suggesting a high skill ceiling. Unclear effect.

AI generation latency

Developers spend approximately 4% of their time waiting on AI generated outputs. This is small but non-trivial, and faster generations would reduce slowdown.

Suboptimal elicitation

Developers use Cursor agents/chat in most AI-allowed issues and sample few tokens. Existing literature finding positive speedup also uses few tokens. Unused elicitation strategies could improve AI reliability. Unclear effect.

Non-frontier model usage

Developers primarily used Claude 3.7 Sonnet (25%, 34%), Claude 3.5 Sonnet (23%), and GPT-40 (11%). These were frontier models for Feb-June 2025. Unlikely to contribute to slowdown.