Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for AI Review Engine
Inspiration
AI systems can generate more information than people have time to verify. The next challenge is therefore not only helping AI produce answers, but helping humans review, verify and understand those answers.
We did not want to build another chatbot. We wanted to build a reviewer:
AI generates. AI Review Engine verifies.
AI Review Engine compares an AI-generated response with the original user request and supplied documentation. It identifies missing requirements, documentation conflicts and claims that lack support, then connects every finding to an inspectable evidence trail.
What it does
The reviewer receives three inputs:
- User Request — what the person actually asked for.
- Documentation — the source material, specification or instructions.
- AI Response — the answer being evaluated.
It produces an evidence-backed report containing:
- missing requirements;
- conflicts with the supplied documentation;
- claims unsupported by the supplied sources;
- requirements already covered;
- evidence references with source, line and quotation;
- a deterministic completeness score;
- suggested areas for human verification.
Findings are verification leads, not verdicts. An unsupported claim is not automatically false, and the confidence indicators are deterministic heuristics rather than calibrated probabilities. Human review remains part of the workflow.
How we built it
We built the MVP as a local, deterministic TypeScript review engine with a React interface. Requirement extraction, normalization, deduplication, evidence matching, conflict detection, stable finding IDs and scoring are separated from the presentation layer.
The interface includes two prepared developer-focused demonstrations:
- reviewing an API implementation against its request and specification;
- reviewing a README against project requirements and documentation.
Users can also edit all three inputs and run their own review. Results can be filtered by missing, conflicting, unsupported and covered requirements, while every finding can be expanded to inspect its supporting evidence.
The prototype runs without accounts, a database, API keys or browser-side model calls. This keeps the demonstration reproducible and prevents private review logic or credentials from being exposed.
How we used GPT-5.6
GPT-5.6 helped us shape the product and reasoning architecture. We used it to:
- turn a broad concept of “AI reviewing AI” into a focused developer-tool MVP;
- define the three-input review workflow;
- distinguish missing information, conflicting information and unsupported claims;
- design cautious language that does not present heuristic findings as facts;
- establish the human-review boundaries;
- design the product narrative, interface hierarchy and demonstration scenarios.
GPT-5.6 supported product, architecture and design decisions. The submitted prototype does not make runtime model calls.
How we used Codex
Codex accelerated the engineering workflow from repository inspection to the verified build. It helped us:
- improve requirement extraction and deduplication;
- create stable, content-derived finding IDs;
- refine detection of HTTP-method, dependency, command, platform and Node.js-version conflicts;
- reduce duplicate findings and false positives;
- ensure documentation conflicts take precedence over redundant unsupported-claim findings;
- make scoring deterministic and resistant to repeated evidence;
- implement complete source, line and quotation references;
- improve accessibility, keyboard navigation, focus management and mobile responsiveness;
- add tests for both prepared demonstrations and important edge cases;
- update the README and Build Week changelog;
- run linting, tests and the production build.
Codex also helped us document the key engineering and product decisions instead of treating the implementation as a black box.
Build Week contribution
Before Build Week, we had experience with documentation analysis, OSINT workflows and earlier review concepts developed inside our broader research ecosystem.
During Build Week, we turned those ideas into a new, standalone and publicly demonstrable product:
- a dedicated AI Review Engine repository;
- the three-input review architecture;
- the deterministic evidence-matching engine;
- structured finding categories;
- stable scoring and content-derived IDs;
- expandable evidence trails;
- two reproducible developer demonstrations;
- automated tests;
- an accessible, responsive user interface;
- explicit review boundaries and human-review safeguards.
This separation between prior experience and the newly built contribution is documented in the project README and changelog.
Challenges we faced
The hardest problem was not generating more findings. It was making each finding useful, cautious and explainable.
Naive text matching creates duplicates, mistakes valid paraphrases for omissions and can label an unsupported statement as a proven hallucination. We therefore introduced normalization, content-based deduplication, structured conflict rules and precedence between overlapping finding types.
Another challenge was producing a score that remained stable when evidence was repeated. We designed deterministic scoring based on unique requirements and bounded penalties rather than counting every matched phrase.
Finally, we had to communicate uncertainty honestly. The engine only knows the materials supplied to it. It does not search an external corpus, and incomplete documentation can still produce false positives or missed findings.
What we learned
We learned that review systems require a different design philosophy from generation systems.
A generator is rewarded for producing a useful answer. A reviewer must additionally show:
- what it checked;
- which evidence it used;
- what remains uncertain;
- why a human should inspect a particular issue.
That makes traceability and epistemic restraint core product features—not disclaimers added at the end.
What's next
The current MVP deliberately focuses on developer workflows. The same review architecture can later support specialized modules such as:
- Documentation Review;
- Research Review;
- Policy Review;
- Contract Review;
- Instruction Consistency Review;
- AI Output Audit.
The long-term vision is not a collection of unrelated products. It is one review infrastructure with multiple contexts and interfaces:
Don't ask AI only to answer. Ask it to review the answer.
Log in or sign up for Devpost to join the conversation.