CAT — Counterfactual Acceptance Testing

Your tests passed. Would the wrong product have passed too?

Inspiration

AI coding agents are getting better at generating software. But passing tests doesn't necessarily mean the software behaves as intended.

A coding agent could implement a tenant-data export that incorrectly allows one customer to access another customer's data. If the acceptance suite only tests successful exports, the implementation might pass every test.

We built CAT to address this gap.

Instead of only asking whether the implementation passes its tests, CAT asks whether those tests would reject a plausible implementation of the wrong behavior.

What it does

CAT is an acceptance-evidence analysis tool for AI-generated and human-written software.

It examines the relationship between approved behavior, existing assertions and plausible counterfactual implementations.

Its workflow:

Start from an approved behavioral contract.
Identify a plausible incorrect implementation that the existing suite might accept.
Find a distinguishing observable outcome that the original suite missed.
Propose an additional acceptance assertion.
Execute the same scenario against known-good, equivalent and incorrect controls.
Produce source-bound execution receipts and explain whether the assertion improves discrimination without rejecting legitimate behavior.

CAT can return unresolved or unsupported outcomes when the evidence does not justify a stronger conclusion.

Our working demonstration

Our primary demonstration examines tenant-scoped data exports.

The original acceptance suite passes an implementation that incorrectly permits cross-tenant access.

CAT's bounded Oracle Repair workflow evaluates the missing assertion:

response = DENY

It runs the original suite and proposed assertion against four controlled implementations.

Results:

Control	CAT decision
Correct implementation	ACCEPT
Behaviorally equivalent implementation	ACCEPT
Incorrect survivor	REJECT
Already-rejected implementation	REJECT

In this four-control fixture, false acceptance decreases from 1/2 to 0/2, while false rejection remains 0/2.

We also implemented a separate synthetic duplicate-operation idempotency example demonstrating the same verification approach.

These are bounded, self-owned fixture results, not a general accuracy benchmark or proof of software correctness.

How we built it

CAT combines:

TypeScript and Node.js for the analysis engine, CLI and controlled fixture execution.
Source-bound behavioral contracts, scenario inputs and assertion validation.
Repeatable Node child-process execution against four implementation controls.
SHA-256-based evidence identity and machine-readable execution receipts.
Deterministic reviewer scorecards comparing original and repaired acceptance results.
A browser-based Evidence Console presenting test discrimination and supporting evidence.
Nebius Token Factory integration preparation for NVIDIA Nemotron 3 Super.

The Evidence Console is deployed as a static viewer on Cloudflare Pages. It presents recorded evidence; it does not perform live inference in the browser.

NVIDIA Nemotron and Nebius Token Factory

CAT prepares a constrained observation-proposal request using an approved behavior contract and original acceptance cases. NVIDIA Nemotron is intended to propose a distinguishing observable outcome; CAT independently validates any complete proposal against its controlled implementations.

We executed one genuine inference request through Nebius Token Factory using:

Model: nvidia/nemotron-3-super-120b-a12b

The provider returned HTTP 200 with 1,032 reported tokens, one request and no retries.

The model consumed its 192-token completion allowance entirely on reasoning, returning finish_reason=length. CAT rejected the incomplete response rather than treating it as a validated observation proposal.

This demonstrates genuine provider execution and strict response validation, but not a successful model-generated repair. The working four-control repair demonstrated elsewhere was generated and evaluated locally, independently of this incomplete inference.

The video includes a clearly labeled post-recording evidence sequence documenting the Nebius request.

Challenges we encountered

Our biggest technical challenge was determining whether acceptance evidence was genuinely discriminating, instead of merely looking convincing.

We needed to preserve legitimate alternative implementations while rejecting incorrect ones, and keep evidence bound to exact source identities.

We also encountered infrastructure restrictions affecting Docker evaluation, localhost requests and Windows junction tests.

The Nemotron inference revealed another important issue: reasoning models need adequate output-token headroom to return usable structured results.

Accomplishments

Built executable Oracle Repair demonstrations for tenant-scoped exports and synthetic idempotency.
Preserved two positive controls while rejecting a previously surviving incorrect implementation.
Produced deterministic, source-bound execution evidence and reviewer scorecards.
Built and publicly hosted the CAT Evidence Console.
Integrated preparation and strict offline validation for NVIDIA Nemotron observation proposals.
Executed and documented a genuine Nebius Token Factory inference request.
Produced a public demonstration video under three minutes.

What we learned

A successful test run is evidence about the tests that were executed, not proof that the product satisfies its intended behavior.

Likewise, a successful AI API request is not proof that the model generated a valid or useful result.

CAT is designed around that distinction: claims must be supported by observable, reproducible evidence.

What's next

Our next milestones are to complete the model-assisted observation loop with a validated Nemotron proposal, strengthen isolated execution, test against independently sourced repositories and expand the system into a practical acceptance firewall for coding agents.

Current limitations

Our public Evidence Console is an earlier static snapshot. The newest local scorecards and Nebius evidence are separate from that deployment.

The four-control fixtures use Node child processes rather than Docker security isolation. The public historical oracle remains WEAK, P03 evidence is incomplete and P04 is omitted.

The latest full local suite reported 312 passes, 37 failures and 3 skips, including environment and clean-tree restrictions. Full project acceptance is not claimed.

CAT is an evidence-evaluation system, not a proof of general software correctness.