Quorum — Devpost Project Story

Paste the section below into the Project Story field. Fill the Built with tags and Try it out links at the bottom. Everything here is original, written for this project.

Inspiration

We kept running into the same problem: the internet is full of confident claims — in headlines, in social posts, in pitch decks — and almost none of them come with a way to check whether they're actually true. Modern AI made this worse, not better. Chatbots will answer anything with total confidence, and they never tell you when they're wrong.

We wanted to build the opposite of a confident chatbot: a system that treats a claim like a question to be investigated, shows its evidence, reaches a clear verdict — and, most importantly, is honest about how much you should trust that verdict.

What it does

Quorum is a claim-verification engine. You give it a claim or a question, and it:

1. Plans the investigation — an NVIDIA Nemotron model breaks the claim into focused sub-questions, deliberately including angles that could support it and angles that could refute it.
2. Deploys a swarm of grounded agents that run in parallel, each one searching the live web with Tavily and grounding its finding in real sources.
3. Reaches a verdict — Supported, Uncertain, or Refuted — with an inline-cited report and an overall confidence score.
4. Proves the confidence is honest — a calibration layer measures whether the scores are trustworthy, so the number actually means something.

The result is a report where every statement links straight to its source, topped by a verdict and a confidence you can rely on.

How we built it

• Models on Nebius Token Factory. Every model call runs through Nebius Token Factory's OpenAI-compatible API. We use NVIDIA Nemotron in tiers: Nemotron Ultra for the heavy reasoning (planning and the final verdict) and smaller, faster Nemotron models for the high-volume worker calls, so the app stays responsive and credit-efficient.
• Parallel fan-out. The worker agents run concurrently, designed to scale out as Nebius Serverless Jobs, with a thread-pool executor for local runs. A single orchestrator handles retries, timeouts, and graceful fallback so one failed worker never sinks the mission.
• Grounding with Tavily. Each worker issues a live Tavily search and returns findings with citations, which the synthesizer dedupes into a single numbered source list.
• Served on Nebius. The whole app is a FastAPI service — containerized and served from a Nebius Serverless Endpoint — with a streaming (Server-Sent Events) UI so you watch the swarm complete live.
• Front end. A lightweight NVIDIA-inspired interface (vanilla JS + GSAP) shows the mission plan filling in, a color-coded verdict badge, an animated confidence meter, and clickable citations.
• The calibration layer. We run Quorum against a labeled evaluation set and compute the Expected Calibration Error and Brier score, then plot a reliability diagram. ECE is:

$$\text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n}\,\bigl|\,\operatorname{acc}(B_m) - \operatorname{conf}(B_m)\,\bigr|$$

where each bin \(B_m\) groups predictions by confidence and we measure the gap between stated confidence and actual accuracy.

Challenges we ran into

• Making the confidence mean something. Anyone can print a percentage; proving it's calibrated is the hard part. Building the eval set and the reliability tooling was where most of the "real" engineering went.
• Structured output from smaller models. Getting strict JSON (plans, findings, verdicts) reliably meant writing tolerant parsers with sensible fallbacks so a malformed response degrades gracefully instead of crashing the run.
• Orchestrating a parallel swarm with retries and timeouts, while still streaming live progress to the UI, took careful threading and a clean event contract.
• Standing out. "AI research agent" is a crowded space, so we deliberately narrowed Quorum to claim verification and made trust — not just answers — the entire point.

Accomplishments that we're proud of

• A complete, end-to-end product: plan → parallel grounded swarm → cited verdict → live UI, all on the Nebius + NVIDIA stack.
• A calibration layer that turns "trust me" into a measured, defensible claim.
• A design that uses the whole stack — Token Factory, tiered Nemotron, Serverless Jobs, and Serverless Endpoints — rather than a single API call.

What we learned

• How to architect a tiered multi-agent system that's both cost-aware and fast on Nebius.
• That calibration (ECE, Brier, reliability diagrams) is what separates a demo from something you'd actually trust.
• How far you can push open NVIDIA Nemotron models for real agentic reasoning on independent infrastructure.

What's next for Quorum

• Deeper adversarial verification (explicit proponent/skeptic debate before the verdict).
• Domain packs (news, science, finance, compliance) with tuned evaluation sets.
• Continuous calibration monitoring so the confidence stays honest as the world changes.

Built With

• css
• docker
• fastapi
• gsap
• html
• javascript
• matplotlib
• nebius-ai-cloud
• nebius-serverless-endpoints
• nebius-serverless-jobs
• nebius-token-factory
• nemotron-nano
• nemotron-ultra
• nvidia-nemotron
• ollama
• openai-sdk
• python
• tavily
• uvicorn