Spec Court

An AI-powered courtroom that cross-examines code against its specification and delivers evidence-backed verdicts.

Comment

Inspiration

As more organizations rely on software to handle sensitive data, compliance and security requirements are becoming increasingly important. Yet code can appear to work while quietly missing critical requirements: storing data it should not store, omitting validation, weakening access controls, or introducing behavior that was never approved.

A common problem is asking the same AI system that wrote the code to decide whether the code followed the specification. That is effectively asking the defendant to judge its own case.

We wanted to explore a different idea: what if code could be brought before a digital court?

Spec Court treats the project specification as The Law, the submitted code as The Defendant, and each testable requirement as a Charge. An independent AI judge proposes a ruling, while server-side evidence guards verify that the ruling is supported by the actual source code.

What we built

Spec Court analyzes one code file against a product specification or PRD and produces an evidence-backed verdict for every requirement:

  • GUILTY: the code violates the requirement.
  • ACQUITTED: the code satisfies the requirement.
  • INSUFFICIENT EVIDENCE: the requirement cannot be reliably determined.

Each ruling includes source-line evidence, reasoning, and a suggested Sentence: a clear fix that can be sent back to a coding agent or developer.

The system also supports:

  • Contempt of Spec: detects behavior that was not requested.
  • Vague Law: flags requirements that are too ambiguous or untestable.
  • Appeals: reruns the case against revised code while preserving charge IDs.
  • Court Records: downloadable Markdown reports for reviews, pull requests, or compliance documentation.

How we built it

Spec Court is a lightweight web application designed to be easy to run and audit:

  • A vanilla JavaScript frontend provides the courtroom interface.
  • An Express server handles validation, provider communication, and evidence verification.
  • Plain fetch adapters support OpenAI, Anthropic, and Gemini.
  • Structured JSON output keeps the judge's response predictable.
  • Server-side guards verify cited quotes against the submitted specification and code.
  • Unsupported or fabricated evidence is downgraded to INSUFFICIENT EVIDENCE.
  • Automated Node and Playwright tests cover validation, API behavior, evidence handling, appeals, accessibility, responsive layouts, and XSS safety.

The provider key stays on the server and is never sent to the browser. The application does not require a database, account system, or persistent storage.

What we learned

We learned that an AI-generated answer is not enough for a trustworthy review. The result needs to be constrained, checked, and tied to verifiable evidence.

Some of the most important safeguards are simple but powerful:

  • Require a strict output schema.
  • Verify every cited quote against the original input.
  • Correct nearby line-number mistakes when possible.
  • Downgrade unsupported claims instead of presenting them as facts.
  • Treat specifications as potentially ambiguous inputs that may need clarification.
  • Keep provider credentials and proprietary code on the server side.

We also learned that compliance is not only about finding violations. A useful review must explain why something failed and provide a practical path to fix it.

Challenges we faced

The biggest challenge was balancing AI flexibility with reliable behavior. Models may return malformed JSON, invent evidence, cite the wrong line, or make confident claims about requirements that cannot be observed from one file.

We addressed this with:

  • Schema validation and one controlled retry.
  • Evidence matching against the submitted source.
  • Line-number snapping for nearby citations.
  • Explicit handling for absent code and insufficient evidence.
  • Stable charge IDs for appeals.
  • Input-size limits and prompt-injection-resistant framing.
  • A comprehensive offline test suite so the core behavior can be validated without depending on live model output.

Another challenge was presenting complex technical findings in a way that feels understandable. The courtroom metaphor helped turn abstract compliance results into a clear workflow: the law, the defendant, the evidence, the verdict, and the sentence.

What's next

Future versions could expand Spec Court to:

  • Review multiple files and repositories.
  • Connect directly to pull requests and CI pipelines.
  • Add specialized courts for privacy, security, accessibility, and regulatory frameworks.
  • Support organization-specific policies and controls.
  • Preserve signed audit records for compliance workflows.
  • Combine static analysis, tests, and AI reasoning into a unified case file.

Spec Court is not a replacement for security professionals, compliance teams, or tests. It is a transparent review layer that helps teams catch requirement gaps earlier and gives developers evidence-backed feedback they can act on.

Built With

Share this project:

Updates

Submission history