AI in Software Architecture
Assessing LLM 'Design Awareness': A New Benchmark for Enterprise Code Quality
This study provides the first empirical benchmark on whether Large Language Models (LLMs) can understand and detect violations of fundamental SOLID design principles. The findings reveal a critical gap between models that generate functional code and those that produce maintainable, architecturally sound software, a crucial distinction for long-term enterprise system health.
The Executive Impact: Is Your AI Architecturally Competent?
An AI assistant that generates functionally correct but architecturally flawed code is a source of hidden technical debt. This research demonstrates that an LLM's ability to reason about SOLID principles serves as a vital proxy for its "design awareness." Deploying architecturally-naive AI tools risks creating systems that are difficult to maintain, scale, and extend, directly impacting long-term total cost of ownership.
99.7%
Peak Detection Accuracy
37%
Manual Review Overhead
240
Validated Code Scenarios
Deep Analysis & Enterprise Applications
The study systematically evaluates four LLMs across four languages and four distinct prompting strategies. The results highlight that success is not about finding a single "best" model, but about matching the right model and prompt to the specific design context and code complexity.
A stark performance hierarchy exists among models. GPT-4o Mini is the decisive top performer, achieving high F1-scores for principles like Single Responsibility (99.7%) and Open/Closed (74.5%). Qwen2.5-Coder-32B is a distant second, while CodeLlama-70B and DeepSeek-33B struggle significantly, especially with nuanced principles like Dependency Inversion (DIP), where they are effectively unable to provide useful detections.
Prompt engineering has a dramatic impact, but no single strategy is universally superior. A deliberative ENSEMBLE prompt excels at detecting Open/Closed Principle violations (75.7% F1-score). In contrast, a hint-based EXAMPLE prompt is far better for nuanced violations like Liskov Substitution and Dependency Inversion. The indirect, two-step reasoning SMELL prompt consistently underperformed, showing that asking models to first identify abstract "smells" before mapping them to principles is ineffective.
Detection accuracy is heavily influenced by external factors. Increasing code complexity is the single greatest factor that degrades performance for all models. For OCP, accuracy plummets from 64.8% on easy samples to just 18.0% on hard ones. Furthermore, statically-typed languages like C# and Java provide clearer structural signals for LLMs, leading to higher accuracy compared to the syntactic flexibility of dynamically-typed Python.
Models fail for three primary reasons. Principle Ambiguity: LLMs struggle with abstract principles like DIP and LSP, often defaulting to detecting simpler, more structural violations like SRP. Flawed Reasoning Chains: Prompts requiring implicit, multi-step reasoning increase cognitive load and error propagation. Schema Non-Adherence: Models frequently failed to produce output in the requested JSON format, necessitating extensive manual review and cleaning, a major obstacle for production automation.
Enterprise Process Flow
Benchmark Dataset Creation
→
Tailored Prompt Selection
→
LLM Violation Analysis
→
Automated & Manual Classification
37%
of all model responses deviated from the requested format, requiring manual review and labeling. This highlights a critical reliability gap for fully automated, enterprise-scale code analysis pipelines.
| SOLID Principle | GPT-4o Mini (Top Performer) | Qwen2.5-Coder-32B (Runner-Up) |
|---|---|---|
| SRP (Single Responsibility) | Excellent (99.7% F1). Masters detection of classes with multiple, unrelated responsibilities. | Strong (89.0% F1). Competent, but less consistent than the top performer. |
| OCP (Open/Closed) | Good (74.5% F1). Effectively identifies areas where polymorphism should replace conditional logic. | Moderate (58.8% F1). Shows capability but struggles as complexity increases. |
| LSP (Liskov Substitution) | Weak (44.5% F1). Struggles to identify subtle contract-breaking in subclasses. | Very Weak (14.5% F1). Largely unable to grasp this abstract principle. |
| ISP (Interface Segregation) | Good (71.1% F1). Reliably detects "fat interfaces" that force clients to depend on unused methods. | Moderate (67.1% F1). Performs surprisingly well on this structural principle. |
| DIP (Dependency Inversion) | Very Weak (7.0% F1). Fails to detect dependencies on concrete implementations over abstractions. | Very Weak (10.8% F1). Similar to other models, this principle remains a major challenge. |
Case Study: The Risk of Architecturally-Naive AI
An enterprise development team integrates a new LLM-based coding assistant to accelerate feature delivery. The assistant produces functional code quickly, passing all unit tests. However, the study reveals that this same model has almost zero capability to detect Dependency Inversion Principle (DIP) violations.
As a result, the team's codebase becomes tightly coupled, with high-level business logic directly depending on low-level data access implementations. This makes the system rigid and difficult to test or modify. When a new database technology is introduced, what should have been a simple configuration change becomes a massive, system-wide refactoring effort. The initial velocity gain is erased by crippling long-term technical debt, all because the AI lacked fundamental "design awareness." This highlights the need to benchmark and select AI tools based on architectural competence, not just functional output.
Calculate Your Potential ROI
Estimate the annual savings and reclaimed hours by implementing an architecturally-aware AI strategy to reduce technical debt and improve code quality across your development teams.
Estimated Annual Savings
$0
Developer Hours Reclaimed
0
Your Implementation Roadmap
Deploying architecturally-aware AI is a strategic initiative. Our phased approach ensures a smooth transition from evaluation to full-scale integration, maximizing code quality and developer productivity.
Phase 1: Tooling Benchmark & Baselining
Conduct a targeted evaluation of LLM-based coding assistants against your organization's specific codebases and architectural standards. Establish baseline metrics for code quality and technical debt.
Phase 2: Pilot Program & Prompt Engineering
Deploy the top-performing model with a pilot group of developers. Develop and refine a library of custom prompts tailored to enforce your specific design patterns and SOLID principles.
Phase 3: CI/CD Integration & Policy Enforcement
Integrate automated architectural checks into your CI/CD pipeline. Use the LLM to flag SOLID violations in pull requests, providing actionable feedback to developers before code is merged.
Phase 4: Scale, Monitor & Refine
Roll out the validated AI tools and workflows to all development teams. Continuously monitor performance, track code quality metrics, and refine prompts based on evolving architectural needs.
Unlock Architecturally Sound AI
Don't let hidden technical debt compromise your software investments. Schedule a strategy session to discuss how to benchmark, select, and integrate AI tools that understand the principles of great software design.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat