RouteWitness

An evidence-backed release twin where GitLab Duo hypothesizes accessibility risk and deterministic policy decides whether software may ship.

Comment

Inspiration

A route planner can compile cleanly, pass normal tests, and report no automated page-level accessibility violations while a small data or algorithm change silently removes the only usable journey for a wheelchair user. RouteWitness asks the release question that ordinary CI misses:

Did this change remove the only usable route for any supported mobility profile, and can we prove the answer before production?

What it does

RouteWitness is a supervised accessibility release twin for GitLab. A GitLab Duo custom flow produces structured, falsifiable risk hypotheses from a merge-request change. Deterministic TypeScript policy code—not an LLM—then compares the baseline and candidate route graphs across protected mobility scenarios, checks four hard invariants, and returns PASS or BLOCK.

When a regression exists, a reducer isolates the smallest journey that proves it: origin, destination, mobility profile, changed edge, and violated invariant. The workflow emits a machine-readable Release Receipt binding the commit, policy hash, evidence status, and SHA-256 integrity checksum. A human maintainer owns the production decision.

How we built it

RouteWitness extends the MIT-licensed AccessGraph project under Path B. New work includes:

  • a pure TypeScript impact classifier, counterfactual verifier, invariant engine, counterexample reducer, and receipt generator;
  • an MCP JSON-RPC server exposing schema-validated route-analysis tools;
  • a GitLab Duo custom flow with read-only analysis and explicit prompt-injection boundaries;
  • a ten-stage GitLab CI design spanning plan, create, verify, package, secure, release, configure, monitor, and govern;
  • a non-root OCI container plus Google Cloud Run candidate-deployment and replay scripts;
  • a judge-facing evidence board that separates measured facts from configured or unavailable integrations.

The central architecture rule is simple: AI investigates; deterministic policy decides. Duo may explain why a change looks dangerous, but only reproducible code can issue the release verdict.

Challenges we ran into

The hardest problem was evidence honesty. A configured pipeline is not an executed pipeline, a checksum is not a signature, and an automated Axe scan is not a full accessibility certification. We built fail-closed evidence generation so missing deployment metadata, OCI digests, candidate replays, or policy inputs remain explicitly NOT_MEASURED instead of becoming optimistic claims.

GitLab.com also blocked pipeline scheduling for this account behind identity verification before any job ran. We registered a locked project runner and preserved the visible failed pipeline history, but we do not claim remote CI or remote Duo execution. The repository contains the full integration and local verification evidence; the limitation is disclosed here and in the demo.

Accomplishments

  • A memorable separation of responsibilities: probabilistic hypothesis generation with deterministic release authority.
  • Minimal counterexamples instead of unreadable failure dumps.
  • 86/86 unit tests passed.
  • 50/50 Playwright executions passed across five responsive viewports.
  • Zero automated Axe violations in the tested states.
  • 82.31% statement, 79.54% branch, and 90.62% function coverage.
  • Zero production dependency vulnerabilities and zero detected repository secrets.
  • A live, public evidence board whose claims are bounded by machine-readable status fields.

What we learned

The strongest agentic systems are not those that give a model unrestricted authority. They create a clean boundary between judgment and proof. Agents are excellent at generating hypotheses and explaining context; safety-critical release decisions need deterministic, replayable invariants with an audit trail.

What's next

The next production step is to complete GitLab identity verification, run the real Duo flow and GitLab CI jobs, deploy the candidate revision to Cloud Run with zero traffic, replay the witness against that immutable revision, and record the resulting authoritative remote artifacts. We would then validate with real mobility users and campus accessibility teams before any operational claim.

Links

Disclosure

The campus topology and seeded regression are synthetic. SHA-256 is used for integrity, not as a digital signature. Automated accessibility results cover only the tested states. Remote GitLab CI, Duo execution, Cloud Run deployment, and live candidate replay remain unmeasured because GitLab.com required account identity verification before scheduling any job.

Built With

Share this project:

Updates

Submission history