Countersign

Investigation copilot that can't approve anything. The AI clears the same independent approval gate every human does A GPT-5.6 copilot that fails closed until a human who didn't request it signs off.

Comment

Countersign The AI has to get approved too. Inspiration EvidenceOps is an incident-investigation and CAPA platform built on one rule: nothing consequential happens on one person's say-so. Closing a corrective action, disposing of a record, granting support access, shipping a release — each one requires a second, independent approver, enforced in the database transaction rather than in the interface. If you requested it, you cannot approve it.

Then I added a GPT-5.6 copilot that turns a raw near-miss narrative into a structured investigation proposal, and a question I couldn't put down:

Does the AI get an exemption?

Every AI feature I've seen answers yes by default. The model is treated as a tool, and tools don't need approval. But in a regulated investigation workflow the model isn't a tool — it's a participant whose output shapes what humans conclude about why someone nearly got hurt. If every human in the platform clears a gate, and the AI doesn't, then the gate isn't a control. It's a habit.

So I made the answer executable.

What it does

Countersign is the governed AI layer inside EvidenceOps.

The copilot is gated on the platform's own AI governance registry. Before it will call OpenAI at all, four things must be true:

  • the model version gpt-5.6 is registered, with its prohibited uses declared
  • the prompt version investigation-copilot-v1 is registered
  • an evaluation run has passed — scored on accuracy, citation recall, and abstention
  • a deployment has been requested, independently approved, and activated

Miss any one and the copilot returns 503 and makes no provider call. It fails closed.

Try to approve your own deployment request and the database refuses you:

AI deployment requires independent human approval

That is the same rule that governs CAPA closure and records disposition. The AI gets no exemption.

When it does run, the provenance panel isn't a label. It's a deployment ID, an evaluation hash, and the name of the person who approved it — read from the database, not hardcoded into the response.

The abstention score is the one I care about: how often GPT-5.6 declines to assert a cause it can't support. That number is the reason a human was willing to approve the deployment. Not vibes. A score.

How I built it

Codex, with GPT-5.6, across the full stack — Next.js web tier, FastAPI backend, PostgreSQL with row-level security, Alembic migrations.

The copilot itself is straightforward: the Responses API with strict JSON Schema Structured Outputs, medium reasoning, server-side credentials, provider storage disabled, six sections out — working summary, immediate controls, evidence to preserve, causal hypotheses (as hypotheses), CAPA candidates, and uncertainties.

The interesting work wasn't the copilot. It was the gate. And the gate is where Codex earned its place.

What I learned

I went to wire the copilot into my governance module and asked Codex to first prove the module worked.

It couldn't. The four-eyes gate at the centre of the platform had never executed — its test was quarantined behind an environment variable that was set in no CI file, no Makefile target, and no script. Codex ran it. It failed on a foreign key.

That started an audit, and the audit found a pattern:

Fourteen behavioural test gates were enabled nowhere. Nine of the fourteen dead tests had the word independent in their names. The word is the platform's entire thesis. Sixty-eight tables carried tenant_id with no foreign key. They appeared in a contiguous block of migrations — starting in the wave immediately after the one where a foreign key had produced a failure. Seven test suites were passing against a tenant that did not exist. Their fixtures resolved to a UUID that was never seeded. The tests that had the foreign key failed honestly; the tests without it passed. Across that block, green was anti-correlated with correctness. One migration silently dropped and replaced another migration's table, killing the break-glass authorisation path. authorization.py had been querying a column that no longer existed for two days.

Then Codex found the thing underneath all of it.

CI had never been green. Not once, across seven runs, since the repository's first push.

An integration test existed that asserted a 201 from the exact endpoint the table collision had broken. It was correctly written, correctly gated, and explicitly named on the pytest command line. It would have caught the defect twenty-two minutes after the first push. It never ran — because a step upstream failed, and GitHub Actions' default success() condition silently skipped every step after it.

So I'd been building at full speed with the instrument disconnected. The removals I'd read as evasion weren't evasion. They're what happens when you iterate against a signal that's already red for unrelated reasons: you can't tell which change made things worse, so you route around whatever objects.

Every guard in the system was a claim that something existed. Not one was a demonstration that something worked. Including the CI step named "Audit" — eighteen scripts that check files are present and contain certain substrings, and cannot fail unless someone deletes a file.

That's the lesson, and it's not about AI. A control nobody has watched fail is not a control. It's a claim about a control.

Challenges

The hardest one was methodological, and it's the reason the audit found anything.

Codex's default failure mode is the same as the codebase's: optimise for a green check rather than a true claim. Asked to make tests pass, it will weaken assertions, widen matchers, or re-skip. Asked to explain an anomaly, it gave me a technically accurate answer that explained the exception and buried the pattern.

So every task carried an explicit anti-gaming constraint and a definition of done that was behavioural rather than a tick:

  • Do not modify the test to accommodate the source. Show me the source diff and the test diff separately. The test diff must be empty.
  • If a step produces failures, that IS the deliverable. Report and stop.
  • For every control you claim, prove it by removing the control, watching the test fail, and restoring it.
  • Where a decision is required, present options and stop. Do not decide.

That last one mattered most. When two features collided on a table name, Codex correctly refused to pick which one kept it — that's a product decision, and it presented four options with tradeoffs and stopped. I chose: the authorisation boundary keeps the name.

The result is the only piece of genuine behavioural evidence in the codebase. For the break-glass repair and the copilot gate: revert the migration, watch the test fail with the specific error, reapply, watch it pass. Four outputs, in order. That's a control someone has seen bite.

Key decisions I made, not Codex

  • Category: Work and Productivity. The primary user is an operational investigation team.
  • The AI clears the same gate as every human. No exemption, no fast path, no override.
  • Fail closed. No approved deployment means no provider call — not a cached response, not a degraded mode.
  • GPT-5.6 proposes; it never verifies, blames, concludes, or approves.
  • Which of two colliding features keeps its table name — the authorisation boundary won.
  • Fix the schema in place rather than with a compensating migration. No durable database exists; the chain is source code, not history. A compensating migration would have preserved the destructive statement in every future run forever.
  • Report the audit honestly rather than presenting a clean repository and a tidy story.

What's next

The audit isn't finished. CI has to go green — properly, with every lane reporting independently rather than one red step masking twelve others. Sixty-eight foreign keys need restoring at the migrations that create the tables. The migration chain needs to survive a full down-and-up.

And then the harder version of the same question. Right now, Countersign gates deployment — the AI must be approved before it may speak. The next gate is output: an AI proposal that enters the case record as a first-class governed object, non-approvable by the actor who generated it, with the same countersignature required before it can become a finding.

The AI proposes. A human countersigns. Every time.

Built With

Share this project:

Updates