Cozzo Judge

Inspiration

Businesses are putting voice AI agents on their phone lines, but they pick them from price pages and feature grids. Those don't show whether the agent booked the job, stuck to the script, handed a gas leak to 911, or invented a price when it got stuck. I run Cozzo (cozzo.io), an independent site that benchmarks voice AI agents, and I wanted a neutral way to grade real calls instead of marketing claims.

What it does

A reviewer pastes a call transcript, or uploads the JSON a speech-to-text pipeline already produced, and adds the business's approved script. NVIDIA Nemotron, served by Nebius Token Factory, scores the call on five weighted dimensions:

Task completion (30%) Factual accuracy (25%) Escalation and handoff (20%) Made-up answers (15%) Tone and empathy (10%) The scorecard shows an overall score, a verdict (Pass at 85+, Needs review at 70–84, Fail below 70), and the quotes from the call behind each score. Each quote is checked against the transcript and marked verified or unverified. A critical miss, like a skipped 911 instruction or an invented fee, fails the call even when the weighted score looks fine.

Compare mode: runs the same script against two agents and shows the gap per dimension. A gap of 3 points or less counts as a tie. Scenario pack: scores three synthetic scenarios (dental office, HVAC company, med spa) so a buyer sees a pattern instead of one lucky call. Three-model Nemotron bench: a default live score is two calls. Nemotron 3 Nano (30B) writes a fast one-line risk read with a 15 s timeout. Nemotron 3 Super (120B) writes the rubric score. The card shows both model ids and latencies, and flags when Nano and Super disagree. If Nano fails, Super still scores and the card says triage was skipped. Ask Ultra (appeal): Nemotron 3 Ultra (550B) reads the same call and says whether it agrees with the score, on request. It never changes the number. Fix the prompt: Super proposes a revised agent prompt and replacement agent lines, keeps the caller's lines word for word, then rescores the rewrite and shows the before and after. It's labeled as a rewritten transcript, not a new call. In one live run, a failing Harbor Voice dental call went from 26 to 83 after the fix. Streaming progress: /api/judge streams NDJSON status events, so the UI shows Nano, then Super, then the quote check instead of a spinner. Shareable cards: Copy report gives a markdown report. Copy link opens a /card#… URL with the scorecard encoded in it. There's no database or account. Rate limits: 10 live scorecards per visitor IP per hour and 300 per day overall. A Nano+Super score counts as 2, Ultra as 1, Fix as 2, Compare as 2 and the scenario pack as 6. Counters are in memory per server instance, as the README documents. Model menu: Nemotron 3.5 Lightning and Nemotron 3.5 Super VL can also be picked; those choices skip the Nano step. Mock mode: with no API key, it still demos using stable sample scorecards and a heuristic scorer that's clearly labeled as a preview. All businesses, agents ("Northline Voice", "Harbor Voice") and calls in the demo are fictional and labeled synthetic.

How we built it

Next.js (App Router), React, TypeScript, Tailwind and shadcn/ui, deployed on Vercel. All inference goes through Nebius Token Factory (https://api.tokenfactory.nebius.com/v1, OpenAI-compatible POST /v1/chat/completions), with each Nemotron size given its own job, as the track brief suggests: /api/judge: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B gives the fast risk triage, then nvidia/nemotron-3-super-120b-a12b writes the rubric scorecard. Progress streams as NDJSON. /api/appeal: nvidia/Nemotron-3-Ultra-550b-a55b gives a second opinion on hard calls. /api/repair: Super rewrites the prompt and agent lines, then rescores. /api/compare and the scenario pack: one Super call per transcript. The API key stays on the server and is never sent to the browser. Per-IP and daily caps protect the shared key during judging. lib/rubric is a plain TypeScript module with no UI or network code. It handles transcript normalization (plain text, WebVTT, and STT JSON with utterances/messages/segments/turns), prompt building, JSON parsing, weighted scoring, verdicts and quote verification, so a future evaluator bot can reuse the same rules. Unit tests for the rubric, parser and sample scorecards, plus npm run live-smoke, which runs six synthetic calls through Token Factory and prints scores and latency.

Challenges we ran into

Getting reliable JSON from a reasoning model. The parser strips / blocks and markdown fences, keeps the object that contains dimensions, and retries once on invalid JSON. If response_format: json_object is rejected, it retries without it. Quote verification. Model quotes used Unicode non-breaking hyphens and curly quotes, so exact matches failed. We normalize with NFKC, map dashes and quotes to ASCII and collapse whitespace before matching. A quote the model paraphrased is still shown, marked unverified. Serverless time limits. Route handlers allow up to 120 s and each Token Factory request times out at 55 s, so one live call plus a retry can finish. Deployment setup. We had to wire the Vercel project and environment variables so live mode works in production.

Accomplishments that we're proud of

An end-to-end live product: paste a call and get a quote-backed scorecard, using Nano for triage, Super for scoring and Ultra for appeals. The scorecard doesn't stop at a grade: Fix the prompt shows the score moving after a prompt change. Critical-flag overrides that catch the failures that actually hurt a business, like a missed safety escalation or an invented price, even when the average score looks good. A rubric module that's separate from the UI and can be reused. The demo still works with no key, through mock mode.

What we learned

Reasoning-tuned Nemotron models give useful scorecards, but production use needs defensive parsing and evidence checks. Model choice is a real product lever: Ultra for hard calls, Super as the default, Nano and Lightning for fast batch runs. Showing evidence matters more to buyers than the number itself.

What's next for Cozzo Judge

A paid Voice Agent Evaluator bot built on the same lib/rubric, which tests a business's agent on a schedule and reports regressions. Direct audio input (STT, then scoring) and larger scenario packs per industry. Making Cozzo a vendor-neutral voice agent broker: helping businesses choose, test and deploy the right voice agent, with Cozzo Judge scores as the proof. Scheduled batch scoring through Nebius Serverless Jobs, and a shared store for rate limits.

Built With

• nebius-token-factory
• nemotron-3-nano
• nemotron-3-super
• nemotron-3-ultra
• next.js
• node.js
• nvidia-nemotron
• react
• shadcn-ui
• tailwindcss
• typescript
• vercel