GovernDiff

Test business policy changes before they ship. GovernDiff turns draft rules into deterministic impact previews, showing who is affected, what changed, and why.

Comment

Inspiration

I kept coming back to a simple question:

When a written policy changes, how do we know what will actually change in practice?

Most policy reviews focus on wording. People compare documents, leave comments, and approve a new version. But even a small change—such as raising a receipt threshold—can remove some review cases, introduce new ones, or create ambiguity that should stop publication altogether.

Software teams have tests and diffs before they deploy code. I wanted to explore what a similar review process could look like for business policy.

That idea became GovernDiff. The name combines “governance” and “diff”: a tool for reviewing not only how policy text changed, but what that change would do.

What I built

GovernDiff is a local web application that turns written policy into structured, reviewable rules and tests policy versions against the same synthetic records.

The demo begins with an intentionally vague expense-policy sentence:

“Unusually high lodging expenses require manager review.”

GovernDiff extracts the measurable rules it can support, but it does not invent a number for “unusually high.” Instead, it identifies one blocking ambiguity and prevents the policy from being published.

A human then resolves the ambiguity by setting the lodging threshold to USD 275. GovernDiff analyzes the revised policy again and publishes an immutable baseline, mfs-v1.

The baseline contains six rules and produces 18 review Findings across 48 synthetic expense records.

Next, the user creates candidate v2. The candidate:

  • raises the receipt threshold from USD 100 to USD 150;
  • adds a rideshare manager-review rule for expenses above USD 120 without preapproval.

GovernDiff analyzes the candidate as seven rules and evaluates it against the same records.

The resulting Impact is:

  • 18 baseline Findings;
  • 16 candidate Findings;
  • 3 removed;
  • 1 added;
  • 15 unchanged.

The UI also shows the Rule Diff, affected records, and evidence for individual Findings. After reviewing the Impact, the user approves the current candidate and publishes an immutable mfs-v2, while the original mfs-v1 remains unchanged in the release history.

GovernDiff does not approve, reject, reimburse, pay, or submit expenses. Its Findings are review candidates—not legal, compliance, fraud, accounting, or payment determinations.

How it works

I deliberately separated the system into two boundaries.

The language boundary

GPT-5.6 is used to turn policy language into a constrained, typed rule specification.

Live Mode uses exact gpt-5.6 through the OpenAI Responses API with strict Structured Outputs. The model receives only the synthetic policy text, the target version, and static field and rule catalogs.

It does not receive:

  • CSV rows;
  • expense records;
  • employee data;
  • Findings;
  • Impact results;
  • Evidence;
  • approvals;
  • releases.

Model output is not trusted automatically. The server validates its schema, rule vocabulary, source quotations, proposed rule IDs, policy version, and canonical semantics before the result can enter the application state.

I also performed a bounded real GPT-5.6 route-level verification for all three policy versions:

  1. ambiguous draft — 5 rules and 1 blocking ambiguity;
  2. resolved baseline — 6 rules;
  3. candidate v2 — 7 rules.

All three requests passed, with zero retries and no fourth request.

The decision boundary

GPT-5.6 does not calculate Findings or Impact.

Those are produced by deterministic TypeScript code running locally over the same 48 synthetic records.

The local system owns:

  • policy compilation;
  • rule evaluation;
  • stable Finding identities;
  • removed, added, and unchanged classifications;
  • Rule Diff;
  • Evidence;
  • approval guards;
  • immutable release snapshots.

This separation lets the model handle uncertain language while deterministic code handles results that must be repeatable and auditable.

Sample Mode

I wanted judges and reviewers to be able to test the complete product without needing credentials.

Sample Mode is therefore credential-free, deterministic, and network-free.

Its three analysis assets were captured once with GPT-5.4, then human-reviewed, canonicalized, and frozen. The application clearly labels them as precomputed Sample analysis.

I kept this provenance separate from Live GPT-5.6. A Sample result is never presented as if it came from a Live request.

How I used Codex

I used Codex throughout the project as an engineering collaborator, not just as a code generator.

The project was developed through a series of small implementation gates. For each gate, I defined the accepted scope, reviewed the resulting diff, ran focused and full tests, and explicitly approved Git staging and commits.

Codex helped me:

  • reconcile the requirements into exact TypeScript contracts;
  • build the policy compiler and deterministic runtime;
  • implement stable Finding keys and Impact comparison;
  • design the 13-phase state machine and 12-event reducer;
  • handle stale and superseded analysis requests safely;
  • build the Sample and Live analyzer boundaries;
  • implement the strict Responses API adapter;
  • create regression, boundary, and workflow tests;
  • review privacy, secret handling, and Git history before publication.

One of the practical lessons was that Codex became much more reliable when I gave it narrow file scopes, explicit acceptance criteria, and clear stop conditions. When a planned path did not match the canonical design, the work stopped before any file was changed.

The final audited baseline contains 18 test files and 180 passing tests.

Challenges

Refusing to invent policy

The first challenge was making ambiguity a real product state instead of smoothing it over.

It would have been easy for the model to choose an arbitrary lodging threshold and make the demo look complete. But that would undermine the purpose of the project.

GovernDiff treats missing measurable values as blocking ambiguities and requires a human decision before publication.

Keeping results deterministic

Natural-language extraction benefits from a model, but operational results must be reproducible.

I had to keep the record-level runtime, Impact calculation, Finding identities, and release history outside the model. The same inputs must always produce the same results.

Handling asynchronous requests safely

A user can edit policy text, reset the scenario, change modes, or begin a newer analysis while an earlier request is still running.

The application uses a monotonic request sequence, AbortController, revision checks, token identity, and reducer guards so an old response cannot overwrite newer state.

Protecting the provider boundary

The Live integration had to prove that useful AI analysis did not require sending operational records to the model.

I added strict outbound field allowlists, a server-only API key boundary, store: false, safe error mapping, browser-side response validation, and tests that verify record-derived data never enters the OpenAI request.

What I am proud of

I am especially proud that GovernDiff now has:

  • a complete no-key Sample workflow;
  • a real GPT-5.6 verification for all three canonical policy versions;
  • deterministic Impact results of 18 to 16 Findings;
  • inspectable Evidence for removed and added Findings;
  • approval tied to the current revision, analysis, and Impact;
  • immutable v1-to-v2 release history;
  • no automatic Live-to-Sample fallback;
  • 180 passing automated tests;
  • repeated deterministic full-flow verification;
  • a completed repository and Git-history privacy audit with no secrets or private data detected.

What I learned

The biggest lesson was that an LLM should not own the entire decision pipeline.

GPT-5.6 is valuable where the input is linguistic and uncertain. Deterministic code is valuable where the output must be repeatable, comparable, and reviewable.

I also learned that Structured Outputs are only the beginning. A schema-valid response can still be wrong for the domain. Local source verification, canonicalization, rule-registry checks, and semantic validation are still necessary.

Finally, I learned that honest provenance matters. Sample Mode and Live Mode use different models and serve different purposes, so GovernDiff identifies them separately rather than hiding that distinction.

What's next

GovernDiff currently uses one fixed synthetic expense-policy scenario so the complete workflow remains easy to test.

Possible next steps include:

  • additional policy domains and templates;
  • privacy-reviewed document and spreadsheet import;
  • persistent projects and release history;
  • team-based review roles;
  • exportable Impact reports;
  • multilingual policy authoring;
  • broader browser automation;
  • shared deployment;
  • configurable but validated rule vocabularies.

The principle I would keep is:

Use AI to compile uncertain language. Use deterministic code to calculate impact. Keep publication deliberate.

Built With

Share this project:

Updates