Once4All: Skeleton-Guided SMT Solver Fuzzing with LLM-Synthesized Generators
Revolutionizing SMT Solver Fuzzing with AI-Driven Generators
Discover how LLMs synthesize robust test case generators, enhancing software reliability and bug detection efficiency.
Executive Summary: Unlocking Advanced SMT Solver Validation
ONCE4ALL transforms SMT solver testing by leveraging LLMs to create sophisticated, context-aware test generators. This innovation significantly improves bug detection, especially for evolving and complex solver features.
43
Confirmed Bugs
40
Bugs Fixed by Developers
1
One-Time LLM Interaction
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
LLM-Assisted Generator Construction
Skeleton-Guided Mutation
Experimental Evaluation & Impact
LLM-Assisted Generator Construction
- LLMs extract CFGs from documentation for SMT theories, including solver-specific extensions.
- Composable Boolean term generators are synthesized adhering to these grammars.
- Self-correction mechanism ensures syntactic validity of generated terms, dramatically reducing invalid formulas.
LLM-Assisted Generator Construction Process
Extract CFGs from Documentation
→
Synthesize Generators from CFGs
→
Generate Sample Terms
→
Parse & Validate Terms
→
Refine Generators if Invalid
→
Robust Generators Ready
90%
Increase in valid formulas post-self-correction
| Feature | ONCE4ALL | Direct LLM Generation |
|---|---|---|
| Syntactic Validity |
|
|
| Computational Overhead |
|
|
| Adaptation to New Features |
|
|
| Semantic Diversity |
|
|
Skeleton-Guided Mutation
- Skeletons derived from existing formulas populate LLM-synthesized terms.
- Ensures syntactic validity while promoting semantic diversity and deeper solver state exploration.
- Overcomes gaps in SMT-LIB documentation for features like quantifiers.
Skeleton-Guided Mutation Process
Randomly Select Seed Formula
→
Extract Skeleton with <placeholder>
→
Select Generator & Produce Terms
→
Check Sort Compatibility & Adapt Variables
→
Synthesize New Formula
→
Differential Testing
11
Bugs found in solver-specific/new theories
Case Study: cvc5 Finite Field Theory Bug
ONCE4ALL identified a bug in cvc5's finite field theory where the ff.bitsum operator incorrectly ignored coefficient multipliers for constant children. The formula encoded v = v² + 2 mod 3 expecting solutions v = 1 and v = 2, but the solver misinterpreted it as v = v² + 1 due to a faulty implementation. The bug was fixed by ensuring proper weighting of constant terms. This highlights how errors can silently compromise solver correctness in extended theories.
(set-logic QF_FF)
(declare-const v (_ FiniteField 3))
(assert (= v (ff.bitsum (ff.mul v v)
(as ff-1 (_ FiniteField 3)))))
(check-sat)Experimental Evaluation & Impact
- 43 confirmed bugs identified, 40 fixed, across Z3 and cvc5.
- ONCE4ALL consistently outperforms state-of-the-art fuzzers in code coverage and bug-finding ability.
- Skeleton guidance significantly improves effectiveness, yielding more useful test inputs.
| Fuzzer | Unique Known Bugs |
|---|---|
| ONCE4ALL |
|
| OpFuzz |
|
| HistFuzz |
|
| LaST |
|
| TypeFuzz |
|
6
Years a Z3 bug remained latent before ONCE4ALL discovered it.
Calculate Your Potential AI Impact
Estimate the significant time and cost savings your enterprise could achieve by automating SMT solver testing with ONCE4ALL's AI-driven approach.
Annual Savings
$0
Hours Reclaimed Annually
0
Your AI Implementation Roadmap
A clear path to integrating ONCE4ALL and transforming your SMT solver validation process.
Phase 01: Initial Consultation & Needs Assessment
Understanding your current SMT solver testing workflows, challenges, and specific theory requirements to tailor ONCE4ALL for optimal impact.
Phase 02: LLM-Assisted Generator Setup
Automated extraction of CFGs from your solver documentation and synthesis of self-correcting test generators for relevant theories.
Phase 03: Skeleton-Guided Integration & Fuzzing
Integrating ONCE4ALL into your CI/CD pipeline, configuring seed formulas, and initiating continuous, targeted fuzzing campaigns.
Phase 04: Continuous Monitoring & Optimization
Ongoing analysis of fuzzing results, bug reports, and solver coverage, with iterative refinement of generators for maximum efficiency.
Ready to Elevate Your SMT Solver Reliability?
Partner with us to implement ONCE4ALL and secure the foundational components of your formal verification and program analysis systems.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat