Troy

Troy monitors critical ICT vendors daily for DORA Article 28(3) compliance — tracking leadership, legal, and sentiment signals to flag deterioration early, every claim traceable and tamper-proof.

Comment

Project Troy — Submission

Inspiration

Under DORA Article 28, EU financial institutions must continuously oversee their critical ICT vendors. In practice, "continuously" means an annual questionnaire. A vendor fills one in, a compliance team files it, and then nobody looks again for 364 days.

We kept coming back to that gap. A vendor doesn't fail on the day its questionnaire is due. It fails after the CFO leaves, after the layoffs, after the county court judgment nobody circulated internally. All of that is public. None of it reaches the risk register.

The second thing that shaped Troy was less obvious and ended up mattering more. If you build a system that generates adverse assessments about named, identifiable companies, and those assessments are partly written by a language model, you have built a defamation and GDPR exposure with a dashboard on top. A hallucinated claim about a real vendor, circulated inside a bank, is not a bug — it is a legal problem.

So we didn't set out to build a risk score. We set out to build a risk score that a compliance officer could defend in front of a regulator, line by line, including the parts that are uncertain.

What it does

Troy monitors public signals about ICT vendors — legal filings, news volume, leadership changes, headcount movement — and produces a tamper-evident evidence pack that attaches to an Article 28(3) register of information. It is deliberately not a register itself; that distinction is enforced in our CI.

Four things it does that comparable tools don't:

Every observation is hash-chained. Each signal row carries a SHA-256 hash over its own content plus the previous row's hash. The database enforces append-only through both privilege revocation and triggers, so even the owning role cannot silently edit history. Any alteration breaks the chain and the system names the exact broken sequence number.

Verification is a property of the system, not a claim about it. The Evidence screen has a button that re-walks and re-hashes the entire chain live, server-side. It isn't a status light.

Exports are content-addressed and immutable. An auditor asking for the March register gets the bytes issued in March, not a fresh render of March's data. Reproduction is retrieval, never regeneration — there is deliberately no regenerate endpoint anywhere in the API.

GDPR erasure without breaking the chain. Executive names are encrypted under per-subject keys. An erasure request destroys the key; the underlying observation survives, the identifier does not, and the hash chain verifies exactly as before.

The dashboard covers a fleet view, per-vendor detail with citation chips and confidence tiers, an evidence/audit screen, a public methodology page, register export, and a concentration-risk comparison.

How we built it

Two developers, split by domain. Wrik owned capture, the AI pipeline, the scoring engine and the backtest. This repository is the other half — evidence integrity, the API, reporting and register export, security, and the frontend.

The architectural rule we held to throughout: deterministic core, AI at the boundaries.

Two components never contain a model call, permanently — the evidence store and the scoring engine. When someone asks what generates a row hash, the answer has to be SHA-256 over canonical JSON, not "a model helps." That boundary is what makes the audit claim true rather than decorative.

Everything else can use AI, and does. Gemma 4 reads contract PDFs and drafts the ITS register fields, each extracted value carrying the verbatim clause it came from. It generates vendor disambiguation terms — the reason a search for "Plaid" returns fintech news rather than fabric articles. Every Gemma call is optional enrichment: if the API is unavailable, nothing breaks. You just type the fields yourself, which is what you'd be doing today anyway.

Stack: FastAPI and Postgres with SQLAlchemy async, Redis and ARQ for durable jobs, ReportLab for the evidence pack, React with Vite and Tailwind for the dashboard, Clerk for auth, Gemma 4 via Google AI Studio.

Challenges we ran into

Our backtest didn't say what we wanted it to say. We tested against Silicon Valley Bank's collapse with Fifth Third Bancorp as a healthy control. The score crossed threshold on the failure date — coincident, not leading. Worse, the control scored higher than the failure case for six weeks.

That was the hardest moment of the build, because the honest options were both unpleasant: quietly show the chart and let people assume, or publish the weak result under a heading that says "how the score is built — and what it can't see."

We published it. The Methodology page reports one failure case, zero days of lead time, weights that are not calibrated, and a threshold that cannot be derived because the two vendors don't separate. It renders the actual curves so you can see them overlap.

Making the honesty structural rather than aspirational. It's easy to say you won't overclaim and then drift. So we made it mechanical: a CI job greps the entire codebase for language claiming to be an Article 28 register and fails the build if it finds any. The audit metrics render "n/a" and "not measured" rather than a comforting number, because narrative generation isn't wired yet. A dimension with no baseline shows n/a, never a fabricated zero.

Reconciling two independently-built halves. Wrik's pipeline output had a different shape, a different score scale, and three dimensions where our schema expected six. His side was frozen by the time we integrated, so we wrote an adapter that conforms his output to our models — and crucially routes his evidence through the same append_signal path, so the tamper-evidence claim actually covers his data rather than stopping at our boundary.

A long tail of environment problems. PowerShell writing UTF-8 BOMs that broke every Python parser. allow_headers=["*"] being silently invalid alongside allow_credentials=True, which blocked every POST. A requirements.txt missing every database dependency, meaning nobody could run the project from a clean clone. Gemma 4 spending its entire token budget on thinking and returning an empty answer.

Accomplishments that we're proud of

The tamper test. Our smoke test appends signals, then deliberately corrupts a row in the middle of the chain — as the database owner, with triggers disabled — and proves the system detects it and names the exact broken sequence. Then it deletes a row and proves the orphaned successor is caught. Then it crypto-shreds a subject and proves the identifier is unrecoverable while the chain still verifies. All of it runs in CI against real Postgres.

The Methodology page. Most teams will show a polished backtest. We show a weak one, with its limitations enumerated, computed live from calibration files rather than hardcoded. It is the page we were most tempted to fake and the one we're most glad we didn't.

Reproduction is retrieval. Getting the "no regenerate endpoint" rule to hold end-to-end took real discipline — it's genuinely easier to re-render on demand. But an LLM-written narrative regenerated later produces different text, and an auditor comparing two copies of the same report rightly concludes the trail is broken.

Live contract extraction. Gemma 4 reads an ICT services agreement and drafts twenty register fields, each with a confidence score and the verbatim clause it came from. Nothing is saved until an analyst confirms. It turns the register from twenty minutes of typing into a review.

What we learned

Honesty is a feature, and it's load-bearing. We expected the audit trail to be the differentiator. It turned out the differentiator was the willingness to publish numbers that don't flatter us. A risk professional who sees "not measured" where every competitor shows 98% learns something about whether to trust the rest.

Non-determinism has to be architected out, not tested out. The moment a model call sits anywhere in the scoring path, reproducibility is gone and no amount of testing recovers it. Drawing that boundary on day one cost us nothing; drawing it later would have meant a rewrite.

Open weights change the audit argument. With a hosted frontier model you can only ever retrieve a stored artifact — versions deprecate, outputs drift. With an open-weights model at a pinned checkpoint, the model itself is reproducible. Our audit claim is genuinely stronger with Gemma than it would have been with a closed model, which we did not anticipate going in.

Fail-soft has to be tested by breaking things. We only found that a Redis outage would 500 our own chain-verification endpoint — the single most important endpoint in the product — by turning Redis off and running the suite.

What's next for Troy

A real backtest. This is the honest priority and everything else is secondary to it. Thirty to fifty documented vendor failures with matched controls, proper out-of-sample discipline, and a published lead-time distribution with confidence intervals. If the signal is real, we'll have evidence no competitor publishes. If it isn't, we'd rather find out.

Narrative generation and the second audit layer. The citation-resolution and extraction-fidelity numbers currently read "not measured" because narratives aren't generated yet. Wiring the narrator and the entailment audit turns two honest zeros into two real metrics.

Alerting, after calibration and not before. The notification layer is built and deliberately dark. Alerting on an uncalibrated threshold generates noise, not signal, and would undermine the one thing we're actually selling.

Design partners. Three to five mid-size EU financial entities — payment institutions, fund administrators, regional banks — using it free in exchange for real vendor portfolios and honest feedback. Their vendor lists are also the backtest population, which is the loop that makes the whole thing compound.

Legal review before the first paying customer. A product that generates adverse assessments of named companies needs a real legal position, not just good instincts. We have the correction workflow, the confidence tiers, and the crypto-shredding. We don't yet have counsel, and that's the gap to close before this touches a real vendor list.

Built With

Share this project:

Updates