SpecSentry

Turn acceptance criteria into evidence backed browser tests and GitHub-ready bug reports.

Comment

Inspiration

Small dev teams may write / know what their features acceptance criteria is, and may even write them down, but small teams don't always have time to run the comprehensive tests needed to validate those ac's

Manual QA takes time and is often an after thought at the end of a sprint. Traditional browser automation kind of works today, but it needs engineering effort and can become brittle when interfaces change. General browser agents can navigate websites, but they do not always produce the evidence a developer needs to reproduce a problem.

SpecSentry started with one question:

Could AI turn the requirements teams already write into a useful first test pass?

What it does

SpecSentry turns a staging URL, user story and acceptance criteria into an evidence backed browser test.

It:

  • creates a structured test plan
  • lets a human review and approve the plan
  • runs the journey in an isolated Chromium browser
  • records actions, screenshots and browser state
  • returns pass, fail, blocked or inconclusive
  • creates a draft finding with severity, confidence and reproduction steps
  • lets a human edit, approve or reject the finding
  • previews the exact GitHub issue before any external write

The hosted demo is restricted to the bundled Sentry Shop example. Self-hosted deployments can test explicitly approved staging domains.

How I built it

ChatGPT 5.6 Sol in chat and Codex

SpecSentry is a Next.js and TypeScript application deployed on Railway.

The AI workflow has three separate phases:

  1. Planner
    GPT-5.6 Terra converts the specification into structured test steps, expected results, evidence checkpoints, retry limits and stop rules.

  2. Executor
    OpenAI computer use and Playwright run the approved journey in an isolated Chromium session. The executor records what happened but cannot decide whether the test passed or failed.

  3. Evaluator
    A separate model call receives the original criterion, approved plan and persisted evidence. It returns a structured result and creates a finding only when the evidence supports a failure.

Zod validates model and API outputs. SQLite stores runs, findings and review state. Screenshots and action history provide the evidence trail. GitHub export remains behind human approval, preview and explicit confirmation.

Codex was used throughout the build to design the architecture, implement features, write tests, debug the live browser loop, harden security controls and prepare the Railway deployment.

Challenges I ran into

Browser agents are probabilistic. They can click the wrong control, wait too long or misunderstand the visible state.

I reduced this risk with:

  • fixed browser dimensions
  • approved test plans
  • action and runtime limits
  • retry ceilings
  • exact hostname restrictions
  • screenshots after checkpoints
  • partial reports when execution fails

Evidence integrity was another challenge. Every cited screenshot must map to a recorded browser action and a real persisted file. The interface therefore keeps captured evidence separate from AI assessment.

The public deployment also needed strict controls. It blocks arbitrary external sites, private network targets, local files, downloads, pop-ups and unattended GitHub writes.

Accomplishments that I'm proud of

The full workflow works end to end:

  • acceptance criterion to structured plan
  • approved plan to live browser run
  • browser evidence to structured finding
  • human review to GitHub issue preview

A controlled ten-case live evaluation produced:

  • 5/5 expected passes
  • 3/3 seeded failures
  • 1/1 blocked result
  • 1/1 inconclusive result
  • 0 false failures
  • 0 retries
  • 0 missing screenshots
  • 0 off-domain navigation

All three failures produced findings backed by persisted screenshots and recorded actions.

The final application has 71 passing unit and service tests, 16 passing Playwright tests and no production dependency vulnerabilities.

I got the whole thing done while on holiday in Crete with my family (evenings while they slept and cheeky check ins via the mobile app were a lifesaver!)

What I learned

The model is only part of the product.

The useful behaviour came from the controls around it:

  • separate responsibilities for planning, execution and evaluation
  • strict schemas
  • persisted evidence
  • clear stop conditions
  • human approval
  • safe domain restrictions
  • honest handling of blocked and ambiguous criteria

The strongest QA output is not a confident answer. It is a result that shows exactly what happened and gives a developer enough evidence to reproduce it.

What's next for SpecSentry

The next steps are:

  • support authenticated staging accounts
  • generate reusable Playwright tests from approved findings
  • run checks against pull requests
  • compare results between application builds
  • add durable hosted storage for team use
  • support a wider range of staging-site patterns

For Build Week, the scope stayed deliberately narrow: one reliable journey from acceptance criterion to evidence-backed finding.

Built With

Share this project:

Updates

Submission history