ExitCanary

Before you enter, prove you can leave.

Comment

Inspiration

SaaS buying usually asks whether a team can get data in. We wanted to test the harder question before an annual contract or renewal: can the team get its operational structure back out? A vendor can advertise CSV export while silently flattening relationships, dropping custom fields, or changing semantics. Discovering that during migration is too late.

What it does

ExitCanary is a bounded pre-purchase exit drill for SaaS data.

  1. Place a versioned synthetic 33-field CRM canary in a trial.
  2. Request the vendor's advertised CSV, JSON, or ZIP export.
  3. Upload the returned packet and explicitly review proposed field mappings.
  4. Receive a deterministic verdict: EXIT_READY, NOT_EXIT_READY, or NEEDS_REVIEW.

The evaluator runs nine bounded checks and creates a receipt with a SHA-256 digest over the versioned canary, evaluator, normalized packet, confirmed mapping, and assessment. If the evidence or confirmed mapping changes, the digest changes. It is a change detector—not a signature, source attestation, trusted timestamp, or migration guarantee.

How we built it

Codex was our engineering partner: it helped design the threat model, implement the parser and deterministic evaluator, build the interface, create adversarial fixtures, and drive the test and release gates.

GPT-5.6 Sol has one narrow job: after explicit consent, it can propose evidence-referenced semantic mappings for unfamiliar vendor field names. It never decides or changes the verdict. Unresolved or unconfirmed mappings cannot pass.

The application uses strict schemas, bounded CSV/JSON/ZIP parsing, request and expansion limits, origin checks, no evidence persistence, store:false, bounded model output, zero automatic retries, and a transparent deterministic fallback. The public judge build is intentionally keyless and fallback-only; the video separately labels a controlled synthetic live GPT-5.6 run.

Challenges we ran into

The hardest problem was separating “the model found a plausible field” from “the export demonstrably preserved the required business relationship.” We solved it with an application-owned 33-field contract, evidence references, a hard unresolved state, mandatory human confirmation, and a deterministic evaluator.

The second challenge was treating every upload as hostile or simply huge without treating evidence as instructions. We added strict validation, bounded decompression and parsing, no raw HTML rendering, and explicit failure states.

The third challenge was making the public demo honest. Anonymous paid model access needs durable identity and quotas, so the judge deployment does not pretend to expose a free live endpoint.

Accomplishments that we're proud of

We built a real end-to-end evidence path rather than a mocked verdict. The flawed synthetic fixture produces six failures and three passes. The complete fixture produces nine passes and a new digest. The tested source passed 12 Vitest files / 86 tests, five public-preflight tests, seven public-smoke tests, a production build, and a production dependency audit.

The result is deliberately narrow, inspectable, and useful at a real buying moment: before money and data become locked into a system.

What we learned

Semantic reasoning and decision authority should be separate. GPT-5.6 is valuable for mapping vendor-specific names, but similarity is not proof; ambiguity must stop the flow until a human confirms it.

We also learned that a small canary can expose specific loss without proving universal portability, and that a hash binds inputs without attesting their origin. Safe fallback is a product feature, not a hidden failure.

What's next

We would run supervised drills against three CRM trials using only synthetic data, refine the smallest check set that changes a buying decision, and add vendor-specific seed/import instructions plus saved mapping templates. Later versions could support recurring re-tests before renewal, authenticated workspaces with durable quotas, and bounded profiles for support desks, project tools, and HR systems.

90-second judge walkthrough

Open the public demo, run the bundled flawed case, mark the mappings reviewed, and verify NOT_EXIT_READY with six failures. Then use the visibly disclosed simulated fixture swap and re-evaluate to see EXIT_READY with nine passes and a changed digest. The simulated fix only swaps fixtures; it never changes a vendor account, repository, or external system.

Built by Evaldas Vigelis at EV1 Labs · Explore the OpenAI Build Week collection.

Built With

Share this project:

Updates