Logo
Landing page

Countersign

Every claim carries the sentence it came from, a number for how much the model doesn't know, and a test that proves which evidence holds it up.

What inspired us

We kept hitting the same failure in document-analysis tools. You ask a question about a pile of contracts, invoices, or filings, and you get back a confident paragraph. Somewhere inside that paragraph is a claim that matters. There is no way to find out where it came from, no way to know whether the model was sure, and no way to test whether it would survive a comma moving.

That last part is what bothered us most. A flag you can't interrogate isn't evidence. It's a vibe with a percentage next to it.

So we inverted the design. Instead of a model that produces prose and cites afterward, Countersign produces a structured claim first and treats the citation as load-bearing infrastructure. Every insight is a tuple of (subject, relation, object) bound to an exact character span in a source document, with the invariant that

$$\texttt{raw_text}[\,c_{\text{start}} : c_{\text{end}}\,] = \texttt{surface}$$

holds byte-for-byte, for every mention, in every scenario. Nothing is ever recovered by searching the text for a substring after the fact. If we can't point at the characters, the claim doesn't ship.

What it does

Countersign ingests documents, extracts an entity-and-relation graph, scores each relation for two distinct kinds of uncertainty, routes it into one of three review buckets, and lets you ask a question out loud and hear the answer with the cited sentence highlighted as it's spoken.

The part we're proudest of is the ablation endpoint. Point it at any insight, name an attention edge, and it removes that edge and re-runs the model. You get back the confidence delta and whether the routing decision changed.

How we built it

The extraction spine. A BiLSTM-CRF tagger written from scratch in PyTorch handles mention detection, and a hand-rolled graph attention network does relation classification. We deliberately did not use torch-geometric. Its GATConv gives you attention weights, but ablation needs something more specific: mask a named edge at inference time and read per-edge, per-layer attention back out. Doing that through PyG means either mutating edge_index, which is a different counterfactual than the one we're claiming, or forking its forward pass. Writing ~200 lines ourselves was less work than either, and it made the ablation claim honest.

Uncertainty as two numbers, not one. The relation head is evidential. For (K) classes it emits non-negative evidence (e_k), which parameterizes a Dirichlet with (\alpha_k = e_k + 1) and (S = \sum_k \alpha_k). That gives us belief masses (b_k = e_k / S) and, separately,

$$u_{\text{vac}} = \frac{K}{S}, \qquad u_{\text{diss}} = \sum_k \frac{b_k \sum_{j \neq k} b_j \,\mathrm{Bal}(b_j, b_k)}{\sum_{j \neq k} b_j}$$

where (\mathrm{Bal}(b_j, b_k) = 1 - \frac{|b_j - b_k|}{b_j + b_k}).

The distinction does real work. High vacuity means I have not seen enough to say. High dissonance means I have seen plenty and it conflicts. A softmax collapses both into one flat "0.6 confident" and loses the difference. Those two failure modes deserve opposite responses, and in Countersign they get them.

The cascade. Vacuity is what gates the LLM. If the classical model has evidence, it decides locally and cheaply. Only when (u_{\text{vac}} > \tau) does the insight escalate to Nemotron for a second opinion. We tuned (\tau = 0.625) against a target escalation band of 8–20%, landing at 10.34%. So roughly nine in ten decisions never touch an LLM, and the ones that do are exactly the ones our own model admitted it couldn't call.

Routing is a separate composite risk score over graph structure, with weights kept in config and written into the endpoint's own OpenAPI description:

$$R = 0.25\,\text{deg} + 0.40\,\text{cyc} + 0.35\,\text{sev}$$

so the number on a node is one a judge can reconstruct rather than one they have to trust. Cycle participation gets the heaviest weight because in this corpus circular ownership is the signal.

Ablation and fragility. An insight is load-bearing on an edge when

$$\mathbb{1}\big[\,|\Delta c| > 0.10 \;\lor\; r_{\text{before}} \neq r_{\text{after}}\,\big]$$

Everything is rebuilt from the stored raw text through the same parse and the same tagger, never from a cache, so the thing being ablated is provably the thing that produced the citation. Fragility comes from a separate fuzzer with five perturbation families (synonym substitution at 15%, punctuation noise at 10%, and three others), scored as a weighted mean over label flips, relation loss, and (|\Delta c|), with flips weighted highest. The fuzzer is forbidden from writing to the database, and a test asserts stored documents are byte-identical after a run.

Calibration. Confidence is temperature-scaled, (\hat{p} = \mathrm{softmax}(z / T)), fit per tenant and measured with

$$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n}\Big|\mathrm{acc}(B_m) - \mathrm{conf}(B_m)\Big|$$

Recalibration is idempotent under a repeated Idempotency-Key, and the baseline is measured at the org's live temperature rather than a hardcoded (T = 1), so a no-op refit reports a no-op instead of reporting float noise as an improvement.

Voice. Briefings are assembled from templates over structured insight fields and spoken through ElevenLabs. Character-level alignment maps each spoken sentence back to its insight_id, so the feed row lights up with its citation while you hear it. Nothing in the voice path generates text, and that isn't a promise in a README: test_no_generation.py walks the AST of every module on the explanation path and fails the build if a generative SDK import or a raw HTTP client appears, with a file-level carve-out for exactly the two TTS/STT modules and a second test asserting that carve-out never grows to three.

The boring half. FastAPI over Postgres with pgvector and pg_trgm, 13 tables, 28 endpoints, argon2id at time_cost=3, memory_cost=65536, JWT with verified aud/iss, an 8-permission RBAC matrix, keyset pagination so live-arriving insights can't duplicate a page mid-demo, and tenant isolation through a single scoped_query() helper rather than a .where() clause you have to remember in forty places.

What we learned

Two uncertainties beat one. Splitting vacuity from dissonance was the single highest-leverage modeling decision. It's also what made the cascade possible, because "escalate when the model lacks evidence" is a rule you can only write if you measure evidence separately from disagreement.

Our expensive model was worse than our cheap one. We ran the LLM on everything as a baseline. It got 10% accuracy (4 of 40 decisions) against the classical head's 42%. That result is in the evals endpoint, labeled, shipped. It would have been easy to quietly drop the arm. Keeping it is the more interesting claim: the cascade isn't a cost optimization we're rationalizing, it's the configuration that actually performs best.

Selective LLM use is a measurement, not an aspiration. "We only call an LLM when we need to" is a sentence anyone can write. (\rho = 0.1034) on a named corpus at a named threshold is a different kind of statement, and the second one took a tuning script.

Challenges we faced

A stale threshold that looked like 28 broken tests. After retraining the relation head with residual connections, the vacuity distribution compressed. The threshold in config was still the pre-retrain value. That mismatch surfaced as 28 failing tests across three debugging rounds, every one of which looked like a code bug. The threshold was wrong, and worse, the corrected value lived only in a local .env while the committed default and .env.example stayed stale, which meant anyone cloning the repo would have watched the escalation rate jump from 10% to 82.76% and the entire cascade thesis invert in public. We now keep the tuning history in the comment and the two files move together.

Removing the same line three times. A rate-limited FastAPI handler must declare a response: Response parameter. It looks unused. We deleted it from the ablation endpoint, broke the live route, wrote up the lesson, then wrote a fresh module and left it off both voice routes anyway. The lesson didn't fail. We failed to apply it. All eight rate-limited routes now declare it, and the comment above one of them says exactly this.

Our showcase insight wasn't load-bearing. The Meridian insight the demo script points at turned out not to be load-bearing at any of its 24 attention edges. Confirmed twice. The honest options were to pick a different insight or to demo a negative result, and discovering that in rehearsal instead of on stage is the entire argument for rehearsing.

Building the parachute last. The prerecorded voice fallback exists because conference wifi fails. We built it late, which meant the one mitigation for our most likely on-stage failure was the thing least likely to be ready. The fix was two-tier degradation: serve the recording if it exists, otherwise assemble a text-only briefing live from templates with audio_available: false, so there is no input to that endpoint that produces a dead end. Same for the production build, where SERVE_STATIC was read in exactly one place and the bundle was mounted, copied, and never served.