Enterprise AI Analysis

Software Development Tasks & LLM Tools

Novice developers utilize LLM-based tools across most Software Engineering (SE) activities, with a notable exception in Software Project Management (e.g., effort estimation, task prioritization).

The most frequent tasks supported by LLMs include:

• Software Development & Quality Assurance: Information retrieval (55 studies), code generation (45 studies), debugging (49 studies), conceptual understanding (32 studies), code refactoring (10 studies), and testing (various sub-tasks).
• Requirement Engineering & Software Design: Brainstorming (20 studies), problem understanding (14 studies).

The most recurrent LLM tools identified are ChatGPT and GitHub Copilot. While proprietary LLMs dominate, there's an emerging trend in open-source tools like DeepSeek.

Novice Developer Perceptions, Advantages, Challenges & Recommendations

Novice developers hold a mixed perception towards LLM adoption, with a balance of positive and negative aspects.

Perceived Advantages:

• Productivity & Efficiency: Gains in speed (e.g., generating files, completing tasks faster), reduced mental effort, and automation of repetitive tasks.
• Additional Assistance: Support for asking 'dumb questions' without judgment, constant availability, and guidance in novel activities.
• Learning Opportunities: Assistance in learning new coding approaches, concepts, and programming language syntax; use as a tutor; speeding up the learning process.
• Code Quality: Hints for potential improvements and identifying minor errors.

Perceived Challenges & Limitations:

• Novices Not Ready for LLMs: Difficulty evaluating suggestions, struggling with complex contexts, and framing specific prompts. Concerns about low-quality or incorrect solutions.
• Not a Good Fit for Novice Developers: Issues with customising LLMs, potential for uploading proprietary code, misleading AI (sycophancy).
• Losing Learning Opportunities: Missing essential learning experiences, over-reliance, becoming dependent or lazy, lack of understanding for copy-pasted code.

Recommendations:

• Prompt Engineering: Break prompts into small tasks, provide context, ask follow-up questions, and learn best practices.
• Cautious Approach: Adopt an analytical attitude, evaluate AI suggestions critically, use reliable AI vendors, double-check AI suggestions, and treat AI-generated code as a baseline.
• Strategic Use: Utilize LLMs as a learning tool to solidify foundational knowledge, and only after attempting solutions independently.

Study Limitations & Future Research Needs

Limitations identified in the primary studies include:

• Data Collection & Analysis (57.5%): Issues with sampling (e.g., geographic location, number of participants, self-selection bias, misinterpretation of questions, research bias).
• Findings (31.2%): Self-report data, memory bias, generalizability of findings to all LLM tools, and findings becoming outdated quickly due to rapid AI evolution.
• Approach (16.2%): Limitations in task design (confounding variables, non-realistic experimental environments).

Future Research Needs:

Key areas for future investigation include:

• Exploratory Studies (37.5%): Impact of LLMs on game development, early career professional attitudes, gender minorities, organizational settings, pair programming, code reuse, and group dynamics.
• Extension Studies (32.5%): Incorporating other LLM tools, different tasks/projects, and larger participant populations.
• Longitudinal Studies (17.5%): Long-term impact on novice developers' learning, career readiness, and adoption patterns.
• Development & Improvement of Guidelines and LLM Tools (12.8%): Designing new metrics for LLM alignment, specialized GPT models for education, and responsible use guidelines.
• Replication Studies (11.25%): Validating findings in different educational settings and updating with recent LLM versions.

Specific research gaps highlighted include LLMs in domains with privacy restrictions, studies in industry settings, LLMs supporting effort estimation and task prioritisation, and the impact of LLMs on developers' code reading skills and mentorship interactions.