Jury

Inspiration

Founders don't fail because nobody warned them. They fail because nobody checked.

We kept watching capable builders spend months on ideas whose fatal flaw could have been found in an afternoon: a competitor charging a third of the price, a licence nobody mentioned, or a company that tried the same model years ago and now only exists as a Wayback Machine snapshot.

When we asked AI chatbots to "validate a startup idea", we got confident, fluent answers that couldn't be checked. The numbers were invented, there were no sources, and the answer was never "I don't know." The "AI advisor panel" tools were worse. They dressed one model up as five people and called the agreement consensus.

So we gave ourselves one rule: build a validator where a sceptic can check every output. Every number has to trace back to either a URL we actually fetched or arithmetic we actually ran.

What it does

The Jury is a virtual courtroom for a startup idea. Its tagline is Don't build it. Prove it.

1. Pitch. You describe your idea in plain language and choose a target geography and customer segment.
2. Assumption Hearing. The Jury breaks the pitch into single, testable assumptions. It scores each one for how critical it is, how uncertain it is, and whether it can be tested at all. It compares your pitch against a hand-written checklist for your type of business, so the things you never mentioned get flagged as gaps. Nothing is investigated until you confirm the list.
3. Five investigators work in parallel. Each "chair" collects evidence from a different set of sources:
   • Market: competitors, pricing and demand
   • Customer: real pain and willingness to pay
   • Precedent: companies that tried this before, with shutdowns confirmed through Wayback Machine snapshots
   • Dependencies: licences, APIs and suppliers you rely on but don't control
   • Economics: benchmarks that feed a real money model

Every claim must cite a page we actually fetched, and the quoted excerpt must appear on that page. Otherwise the database refuses to store it.

1. Conflict detection. Fixed rules, not an AI model, find real contradictions. The most important is your pitch versus the world. For example, you say ₹499/month and the closest competitor charges ₹149.
2. Cross-examination. It runs only where a real conflict exists. The chairs involved have to bring stronger evidence or back down, and every change of position is recorded.
3. Economics. A typed model calculates margins, customer lifetime value against acquisition cost, and payback. It finds the exact breakpoints where the business stops working, such as "loss-making above ₹38 delivery cost", and ranks which inputs matter most.
4. Verdict. The Jury rules PROCEED, PIVOT or STOP. It can also declare a HUNG JURY when the evidence is too thin. In that case it names the three cheapest experiments that would settle the question, each with a pass/fail line decided in advance.
5. Living ledger. You come back weeks later with real results. The Jury re-runs only what those results affect, saves a new locked version, and explains what changed in one sentence. For example: "pricing_wtp moved uncertain → refuted, which moved price_monthly from ₹499 to ₹249, which moved the break-even delivery cost from ₹38 to ₹19. Verdict changed PROCEED → PIVOT."

How we built it

Architecture: a Next.js 15 frontend, a FastAPI and LangGraph backend, and Supabase (Postgres) as both the database and the live update feed.

• Pipeline (LangGraph): a long-running graph that fans out to all five chairs at once, sends a run to cross-examination only when a conflict exists, and pauses for the founder at the Assumption Hearing. Its state is saved in Postgres, so a run that crashes or pauses picks up where it left off, even weeks later.
• The AI layer: Groq models (Llama 3.3 70B, Llama 3.1 8B, GPT-OSS 120B) accessed through the LiteLLM gateway, with a fallback chain, retries, budget caps and response caching. The models only do language tasks: pulling out assumptions, turning page text into typed claims, cross-examination, and writing the verdict explanation.
• Where AI is deliberately not used:
  • Checking that a source exists and that a quote appears on the page
  • Rating how trustworthy a source is
  • Removing duplicate evidence
  • Detecting conflicts
  • All arithmetic (NumPy, and SciPy's brentq root-finder for breakpoints)
  • Confidence scoring and the verdict itself

An automated test fails the build if any of this decision-making code imports the AI, network or database layers.

• Making open-source models reliable: every structured answer goes through a repair loop. The steps are JSON mode, schema validation, a retry that shows the model its error, a fallback to smaller single-field requests, and finally dropping the claim and logging why. A dropped claim is acceptable; a broken database row is not.
• Retrieval: Brave, Tavily, Exa, Hacker News (through Algolia), Reddit, the Google Play Store and the Wayback Machine, with a query budget for each chair. Pages are read with trafilatura, falling back to Jina Reader. Text is embedded on our own machine with fastembed (bge-small-en-v1.5) and stored in pgvector.
• The Evidence Ledger: 17 Postgres tables, with row-level security on every one. Evidence is locked by the database itself: no role, not even a superuser, can edit or wipe it, yet project and account deletion still work.
• Live boardroom: Supabase Realtime pushes each new evidence row to its chair's column as it lands. A chair appears to "speak" only when it has written a real row.
• Offline mode: every outside service sits behind an interface that has a live version and a recorded-response version. With JURY_OFFLINE=1, the whole pipeline and all 680+ backend tests run with no API keys.
• Published scoring: every formula, weight and threshold is documented in docs/SCORING.md and kept in sync with the code.

Challenges we ran into

• Open-source models are unreliable at structured output. Every chair must produce typed JSON records, which made this our biggest reliability risk. One bug that took two rounds to fix: we ask for a schema in every prompt, so models sometimes repeated an example of it, and our parser accepted that example as the real answer. We now validate every JSON candidate in a response and keep the last one that passes.
• Edge cases that broke the verdict gate. Code review found several ways around it:
  • A NaN confidence score reached PROCEED, because every comparison with NaN is false.
  • An empty list of critical assumptions passed as "all supported".
  • An empty record still scored 40/100, earning credit for finding nothing wrong because it checked nothing.

We fixed each one and now brute-force test the gate over 504 input combinations.

• Security holes when the AI picks which URLs to fetch. Because the model chooses the targets, a single trailing dot (169.254.169.254.) got past our entire blocklist and reached the cloud metadata endpoint. We also closed loopback addresses written in octal or hex, look-alike Unicode hostnames, and a malicious redirect that crashed the fetcher before our redirect check could run.
• Locked evidence versus the right to delete. Making evidence impossible to edit at the database level accidentally made it impossible to delete a project or an account, because of how Postgres locks foreign keys. We narrowed the rule instead of weakening it: nobody can falsify or wipe evidence, but a user can still erase their own project.
• Tests that could never fail. Deliberately mutating the conflict rules left the whole test suite green. Several rules had no test that could actually catch a bug, so we rewrote those tests until they failed when the code was wrong.
• A two-day build window. We had to cut features without cutting integrity, and we wrote down every cut and every deviation from the product spec.

Accomplishments that we're proud of

• Every number is traceable. Each figure the Jury shows comes from a fetched source or a calculation, and you can click through to check it.
• A system that can say "I don't know." Below its evidence thresholds, PROCEED and STOP are impossible to reach, and we test that by brute force.
• The fabrication guard works. 50 out of 50 made-up companies were rejected before reaching the database, and the rules for rating source trust scored 100% on 100 labelled URLs.
• The founder's pitch is treated as claims that can lose. The "your pitch vs the world" rule is the feature that justifies the product.
• The AI never does the judging. Scoring, conflicts, arithmetic and the verdict are plain, testable code, and the formulas are published so anyone can argue with the weights.
• Integrity enforced by the database, not by prompts. No source means no row, evidence can't be edited, the same source can't be counted twice, and every experiment has a pass/fail line set before it runs.
• Runs that survive interruptions. A run can pause for a human, crash, and resume from a brand-new process.
• The whole pipeline runs with zero API keys, which made every phase testable before any credentials existed.
• An honest changelog. Every build phase records its exact test command, the full output, and the bugs code review caught.

What we learned

• The strongest engineering decision was where not to use AI. Language models are excellent at reading and writing, and they shouldn't do arithmetic, detect conflicts, or deliver a verdict.
• Five personas on one model are one opinion. Real disagreement comes from different sources of evidence, not different system prompts.
• Refusing to answer is a feature. A STOP based on thin evidence is as irresponsible as a PROCEED based on thin evidence, and people trust a system more when it says "here's how to find out."
• Enforce guarantees at the lowest layer you can. Prompts can be ignored; database constraints can't.
• Listing bad inputs is not the same as normalising them. Our 24-case URL probe missed the trailing-dot bypass completely, and cleaning up the hostname first is what closed it.
• A test that can't fail is worse than no test. It gives false confidence, so we now deliberately break code to check that our tests notice.
• Write down every deviation from the plan. Drift you record is a decision; drift you don't record is a bug.

What's next for Jury

• A growing precedent library. Every verified precedent will enrich future runs across all users. It is the one asset the product builds up over time.
• Scheduled re-checking of sources. Re-fetch key sources regularly and flag evidence that has changed, such as a competitor raising prices or shutting down.
• Better page reading. Add a headless-browser tier to reach JavaScript-rendered pricing pages.
• Read-only ledger links for investors, turning the case file into a due-diligence document.
• Accelerator and cohort mode: validate many ideas at once with shared coverage reporting.
• Automatic experiment results, pulled from analytics and payment tools instead of logged by hand.
• Regulatory packs by country, to strengthen the Dependencies chair in each jurisdiction.
• Calibrated confidence. Fit the confidence weights to real project outcomes. Today they are a published judgement call, not a fitted result.

Built With

• api
• docker
• fastapi
• langgraph
• llm
• node.js
• python
• react
• supabase