Promise Ledger

Inspiration (An illustrative scenario) A customer has waited three weeks for an audit-export fix, and their security review is on Friday. On Tuesday engineering closes the ticket, and you write back: "Good news, it's shipped!" Ten minutes later: "We still can't see it." The fix is real, merged and in the changelog, but the feature flag was never turned on for their account.

I am the head technical support at Sentry, and anyone in support or customer success knows this gap between "engineering finished" and "this customer can actually use it." Our tools summarize activity well. They don't answer the question that matters before an update goes out: can I truthfully tell this customer we delivered what we promised?

(Please note: this is aersonal hackathon project. Not a Sentry product, not affiliated with or endorsed by Sentry, and no Sentry customer data. The Sentry integration reads only a free demo project with synthetic errors.)

What it does

Promise Ledger is an evidence-first workbench for support engineers and customer-success managers preparing an update for one customer. Pick an account, click Run evidence check, and watch three NVIDIA Nemotron models on Nebius Token Factory work live:

Nemotron 3 Nano triages the sources and flags prompt-injection attempts.

Nemotron 3 Super extracts every commitment (feature, owner, due date, committed or tentative), each backed by exact quotes.

Deterministic rules, not a model, decide the verdict from three separate facts: is it built, enabled for this customer, and verified by this customer?

Nemotron 3 Ultra explains why the evidence disagrees, drafts the customer update and writes a nudge to the internal owner. Every claim must cite source text; anything that adds a date, a promise or a verdict change is rejected.

Two outside evidence sources: Sentry runtime errors: "enabled, but crashing for this customer" lowers a verdict to needs verification. No errors never proves delivery.

Tavily public claims: when the public changelog says "generally available" but this customer can't use it, Promise Ledger flags "publicly GA ≠ usable by this customer" and warns the drafter. It never changes a verdict.

Try four fictional sample accounts (SaaS, healthcare, logistics, banking) or bring your own evidence: paste notes, tickets or Slack threads, confirm the facts Nemotron proposes, and let the rules decide. You edit and approve the update; nothing is ever sent automatically. Live AI is open to everyone with no login, within fair-use limits. All data is synthetic.

How I built it Nebius Token Factory serves all three models through its OpenAI-compatible API: serverless, pay-per-token, no GPUs to run. Models: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B, nvidia/nemotron-3-super-120b-a12b, nvidia/Nemotron-3-Ultra-550b-a55b, called server-side with streaming and JSON output. A full run takes about 20 seconds and costs about a cent.

Token Factory accelerated us because switching or adding a model was a one-line change to the model ID. That let us give each step the right model size and measure all three against the same evaluation set in an afternoon.

Trust boundary: strict schemas; every quote must appear word for word in an authorized source; owners and dates must sit inside those quotes; the reported model must be the expected Nemotron model. Failures are dropped or fall back one item at a time and are always labelled, never silently replaced.

Evidence providers: Tavily Extract on an allowlisted public changelog (no generated answers used), and read-only Sentry issues filtered by customer and feature tags, re-seeded daily by a Vercel cron.

Open judge access at $0: per-visitor rate limits plus a durable $40 lifetime spend cap in Upstash Redis, below our $50 of promo credit. Partners run on free plans.

Stack: React 19, TypeScript, vinext (Vite-based Next.js), Tailwind, Zod, deployed on Vercel. 274 unit and service tests plus production build tests.

AI-assisted development: the first prototype with OpenAI Codex, everything since with Cursor cloud agents. The video narration is a synthetic text-to-speech voice.

Challenges I ran into

Keeping models out of the verdict. Nano routes, Super interprets, Ultra explains, and only after the rules have decided.
Reasoning defaults. With default reasoning, Nano timed out and Ultra ran out of output. reasoning_effort: "none" fixed both; Super keeps reasoning on because extraction needs it.
Exact quotes are hard. At temperature 0, Super sometimes dropped the speaker label ("Maya Chen: ") from quotes, and one ungrounded item used to reject the whole run. A double quote in an error record could break Ultra's JSON. Both fixed; runs now complete reliably.
Untrusted evidence everywhere. Pasted text, fetched pages and error feeds are treated as data, never instructions. A sample ticket with a prompt-injection attempt is flagged and changes nothing.
Keeping judge access (yours) free through December 15. The Token Factory trial ends in October, and paid usage has no hard stop when credits run out. We built our own rate limits and lifetime spend cap so live AI stays safely available for judging.

Accomplishments that I'm proud of

A crisp idea you can see: built ≠ enabled ≠ customer-verified. The verdict changes because the evidence changed, not because someone rewrote a summary.
A live, visible three-model Nemotron pipeline where each model has one job and none decides the outcome.
Measured with real inference: the full pipeline matched 9/9 development and 31/32 frozen held-out cases, and 26/26 Ultra briefs passed our automated checks. (Small synthetic test sets, not a production accuracy claim.)
Five evidence families, each quote-checked, including Sentry runtime errors and Tavily public claims.

What I learned Where do I start!?!

When trust matters, use the LLM as an interpreter whose output is validated, never as the decision-maker.
Match the reasoning setting to the job: triage and writing work well with reasoning off; extraction needs it on.
Guardrails need tuning in both directions: too loose lets wrong citations through, too strict rejects correct ones.
Hackathon logistics (...) are engineering too: credits, trial length and keeping judge access alive.

What's next for Promise Ledger

Feedback from working support engineers and CSMs. We haven't run interviews and don't claim adoption or time savings.
Real connectors (CRM, support desk, feature flags, telemetry, error monitoring) with proper authorization and account mapping.
A fresh, independently written evaluation set and human review of Ultra's drafts.