Inspiration
While experimenting with AI driven test generation, I found that asking an LLM to handle the entire workflow from requirements to test cases and final automation code encountered several problems. Models could generate locators that had never been verified against the real application, create assertions based on assumptions, produce inconsistent automation patterns, and repeatedly spend tokens reasoning about application knowledge that had already been discovered. This led to the core idea behind AQAI. AI should reason about software, but verified evidence should control what becomes executable code. I wanted to explore whether an AI-assisted QA platform could combine the semantic reasoning capabilities of modern language models with deterministic software engineering practices and real browser evidence. That became AQAI.
What it does
AQAI transforms software requirements into verified, executable automated tests through an evidence-first pipeline:
Requirements -> Canonical Test Cases -> UI Discovery -> Verified Evidence -> Confirmed UI Catalog -> GPT-5.6 Reasoning -> Typed Contracts -> Deterministic Java Generation -> Quality Gates -> Test Execution.
Instead of asking an LLM to directly generate arbitrary Selenium code, AQAI first discovers the live application and collects UI evidence. Locator candidates are scored and verified against the real browser. Only confirmed evidence can be promoted into the UI catalog and used by later stages of the pipeline. GPT then performs constrained semantic reasoning over this curated evidence, including page enrichment and structured Page Object contract planning. The resulting contracts are validated before deterministic components generate the final Java Page Objects and TestNG tests. The generated code then passes through compilation, automated code review, live smoke validation, and real browser execution. If AQAI cannot find sufficient verified evidence for an action or assertion, the architecture is designed to report a coverage gap rather than allow AI to invent unsupported automation.
How we built it
AQAI is primarily built with Java 17, Selenium WebDriver, TestNG, Maven, GPT 5.6, Neo4j, Qdrant, and Docker. The platform uses a structured pipeline where different stages exchange typed artifacts rather than relying on a chain of unrestricted natural-language prompts. I intentionally separated probabilistic reasoning from deterministic execution.
GPT-5.6
GPT 5.6 serves as the runtime semantic reasoning layer. It is used for focused tasks where semantic understanding provides value, including:
- semantic page enrichment
- structured Page Object contract planning
The model operates on curated application evidence and produces structured outputs that must pass validation before they can influence generated automation. GPT 5.6 does not directly write the final Selenium implementation used by the demo.
Deterministic AQAI Core
The deterministic platform components remain responsible for
- UI evidence validation
- locator scoring and promotion
- Page Object ownership rules
- Java Page Object generation
- TestNG test generation
- compilation
- generated-code review
- live smoke validation
- final test execution
This separation allows AQAI to benefit from AI reasoning while keeping executable automation reproducible and auditable.
Codex
Codex was used as a repository aware engineering collaborator during the development of the Build Week version of AQAI. It supported architecture enforcement, implementation, typed pipeline refactoring, code review, documentation, debugging, and end ti end workflow stabilization. Architectural rules were defined at the repository level to help ensure that changes preserved the canonical evidence lifecycle and did not introduce alternative paths that could bypass evidence promotion or quality gates. Codex helped me build and evolve the system, while the AQAI runtime remains responsible for executing the final testing workflow.
Knowledge reuse
Another challenge I wanted to address was repeated AI reasoning. Once AQAI has discovered an application, verified its UI evidence, and produced validated semantic knowledge, equivalent future workflows should not always require the model to solve the same problem again. To explore this, I integrated Neo4j and Qdrant as a persistent knowledge layer. AQAI can retrieve previously validated application knowledge and reuse stable enrichment and Page Object contracts when the available evidence supports it.
The Build Week demo compares two execution paths using the same four requirements.
Without knowledge reuse
- 4 GPT 5.6 runtime calls
- 2 page enrichment calls
- 2 POM planning calls
- 2 generated Page Objects
- 4 generated TestNG tests
- 0 generated-code review findings
- successful compilation
- successful live smoke validation
- 4/4 generated tests passed
With validated knowledge reuse
- Neo4j knowledge retrieved
- Qdrant knowledge retrieved
- 2 page enrichment cache hits
- 2 stable POM contracts reused
- 0 new GPT 5.6 runtime calls
- 2 generated Page Objects
- 4 generated TestNG tests
- 4/4 generated tests passed
The key result was:
Same requirements. Same validated result. Four fewer runtime model calls. Knowledge reuse does not bypass AQAIs validation and quality gates. The goal is to avoid unnecessary repeated semantic reasoning while preserving the deterministic execution path.
Challenges we ran into
One of the biggest challenges was defining the boundary between probabilistic AI reasoning and deterministic automation. Giving an LLM complete control over test generation is flexible, but it makes reliability, locator quality, reproducibility, and application state difficult to control.
The solution was to separate the workflow into explicit stages: Discovery -> Evidence -> Reasoning -> Contracts -> Deterministic Generation -> Validation -> Execution
Another challenge was reliable UI discovery in modern single-page applications. A DOM element being present does not necessarily mean that it is stable, relevant to a requirement, or safe to use in generated automation. This required introducing evidence scoring, live browser verification, page ownership, and promotion rules before discovered UI information could become trusted knowledge. Persistent knowledge introduced another challenge: deciding when previously discovered information is stable enough to reuse and when the platform should perform new reasoning. The knowledge layer therefore had to work alongside the evidence lifecycle rather than become a shortcut around it.
Finally, making the complete workflow reproducible required packaging the platform and its dependencies for Docker based execution while keeping browser automation, AI reasoning, knowledge storage, deterministic generation, and validation working as one end to end system.
Accomplishments that we're proud of
The biggest accomplishment is building a complete, working vertical slice that transforms software requirements into verified and executable automated tests.
For the Build Week demo, AQAI successfully
- processed 4 capability-first requirements
- generated 4 canonical test scenarios
- discovered and confirmed 2 application pages
- verified locator evidence against the live application
- generated 2 deterministic Java Page Objects
- generated 4 executable TestNG tests
- completed automated generated-code review with 0 findings
- successfully compiled the generated code
- passed live browser smoke validation
- executed all generated tests successfully with 4/4 tests passing
I am especially proud of demonstrating that AI reasoning and deterministic test automation can work together without giving the language model unrestricted control over executable code. Another key accomplishment is the persistent knowledge layer. In my comparison, the baseline workflow required 4 GPT 5.6 runtime calls and produced 4/4 passing tests. When validated knowledge was available through Neo4j and Qdrant, AQAI reused semantic enrichment and stable Page Object contracts, required 0 new GPT 5.6 runtime calls, and still produced the same 4/4 passing result.
This demonstrated one of the core ideas behind AQAI
- Use AI when new reasoning is required
- Reuse validated knowledge when the problem has already been solved
What we learned
Language models are extremely useful where semantic understanding is required, but many decisions in test automation benefit more from deterministic rules, explicit contracts, and verifiable evidence. I also learned that persistent knowledge can change how an AI system uses models. Instead of treating every execution as a completely new reasoning problem, validated knowledge can become a reusable engineering asset.
This suggests a different architecture for AI assisted software engineering
- Use AI when new reasoning is required
- Reuse knowledge when the problem has already been solved
- Validate both before execution.
Working with Codex also reinforced the value of giving AI coding agents explicit architectural constraints. Repository level rules and clearly defined sources of truth made it easier to use an AI engineering collaborator without allowing the architecture to gradually fragment into parallel decision paths.
What's next for AQAI
The Build Week demo focuses on a complete vertical slice from requirements to executable UI tests.
The next steps for AQAI include expanding the same evidence first architecture into
- broader multiapplication UI discovery
- requirement driven API testing
- evidence based self healing locators
- deeper runtime feedback loops
- cross run quality analysis
- knowledge versioning as applications evolve
- richer retrieval across UI, API, and requirement knowledge
The long term goal is not simply to generate more test code. It is to build an independent AI assisted quality engineering platform that can participate across the testing lifecycle while keeping AI reasoning grounded, reusable, explainable, and separated from deterministic execution.
AI reasons. Evidence validates. Deterministic code executes.
Log in or sign up for Devpost to join the conversation.