Inspiration
Ask a normal RAG chatbot "who supplies NVIDIA?" and it will confidently list companies that never appear in NVIDIA's filing, or that appear only as competitors, lawsuit parties or names in a glossary. For anyone doing supply-chain or exposure research, an answer you can't check is worse than no answer. We wanted a research analyst that refuses to name a company it can't quote.
What it does
Ripple answers "who depends on whom" questions (suppliers, customers, competitors and partners) across 15 US tech companies in three sectors: semiconductors, equipment and hardware.
- Every company it names comes with the exact 10-K sentence and an SEC link that scrolls to that sentence.
- A verifier sub-agent checks each claim against the filing before the answer is shown, and drops anything it can't find.
- Follow-up questions keep context ("which of those also supply AMD?").
- An interactive relationship graph shows 102 filing-backed links between companies. Clicking a line shows its source sentence. Price-based links show how closely each company moves with its sector.
- A Proof page recomputes every metric live, and a blind audit page lets a human grade graph links next to independent ones.
- ElevenLabs voice: ask by microphone, hear the answer read back.
How we built it
- Data: SEC EDGAR 10-Ks and XBRL revenue, plus daily prices from Yahoo cross-checked against Nasdaq, stored in DuckDB.
- Graph:
- We store every sentence in which one filer names another company, with no keyword filter.
- Sonnet 5.5 labels each sentence as supplier, competitor, partner or none, using 1,500 characters of context.
- Company↔company links come only from filings. Company↔sector links come only from prices: beta and correlation against the sector average, with the company itself left out.
- Agent:
- Built with LangChain Deep Agents. Every model call goes through the Nicrron API (Sonnet 5.5 for the agent, gpt-4.1-mini for reranking, text-embedding-3-small for embeddings).
- Tools: graph lookup, sector stats, a demand-shock calculator and Knowledge Base search.
- A verifier sub-agent checks every claim.
- Baseline: plain RAG, using hybrid BM25 + embedding search, reciprocal rank fusion and an LLM reranker.
- PRISM:
- Every call is traced (1,900+ traces under agent
ripple, one session per question). - Knowledge Base proof: all 15 filings are uploaded and assigned to
ripple, and every run'skb_searchstep retrieves from them. - Evaluator proof: we wrote our own evaluator, "Every company is in the source", and set it up to score v1 and v2 on the same questions.
- Root Cause proof: our sentence: plain retrieval reads lists of company names as relationships, because nothing checks who the sentence says is doing what.
- Every call is traced (1,900+ traces under agent
- UI: React + shadcn + react-force-graph, with a FastAPI backend.
The failure → fix → score loop: same 60 questions for every version. Precision and recall are measured against a reference built by reading every filing in full.
| Version | Claims backed by an exact quote | Precision | Recall | Seconds per question |
|---|---|---|---|---|
| Plain RAG | 45% | 78% | 94% | 248 |
| Ripple v1 (gpt-5-mini, keyword graph) | 74% | 91% | 96% | 175 |
| Ripple v2 (Sonnet 5.5, every-mention graph) | 100% | 77% | 96% | 55 |
| Ripple v3 (v2 + strict relationship rule) | 100% | 89% | 92% | 53 |
Paired sign tests:
- v1 vs plain RAG: precision p = 0.020, quote-backing p = 0.006
- v2 vs v1: quote-backing p = 0.016
- v3 vs v2: precision p = 0.019
Challenges we ran into
- Keyword extraction was brittle. Sentences split on "Inc.", keywords matched inside company names, and executive bios and competitor lists became fake links. Worse, it missed links stated without a keyword ("We purchase memory from SK Hynix"). We dropped the keyword gate and labelled every mention.
- Better model, worse precision. Switching to Sonnet gave 100% quote-backing but dropped precision from 91% to 77%: it counted lawsuits, spin-offs and competitor lists as partners. v3's strict rule (only the filer's own current commercial relationship counts) brought it back to 89%, at a cost of 4 points of recall.
- Coverage is limited by the name list. Recall is 91% on companies Ripple knows, but only 31% overall: 130 of 136 misses name companies outside our 28-company list (MediaTek, GlobalFoundries, TI, NXP…). Only 6 are true pipeline errors.
- Independent ground truth is hard. The coverage reference was extracted by Sonnet, the same model family that labels the graph, and our 24 gold answers were written by Claude, so their errors may be correlated. We built a blind human audit page for this, but it's only partly graded so far.
- PRISM platform issues. Evaluator runs complete with 0 results, even for the built-in evaluator, and Root Cause hasn't analysed any traces yet, so our v1 vs v2 score comes from running the same check locally. PRISM's generic trajectory scores also rated the old v1 higher than v2, because they don't measure quote-backing.
- Provider quirks. Through Nicrron, Anthropic models reject forced tool choice, so we parse structured JSON from text. A 60-requests-per-minute rate limit needed retry passes. 2 of 60 baseline runs still errored and were excluded.
- Disclosure: early on we tried TypeSafe's Jev API (outside Nicrron) as a cheap labeller. It was an offline experiment and isn't used anywhere in the agent, the graph or the demo.
Accomplishments that we're proud of
- An answer you can audit in one click: every company links to the sentence that justifies it.
- Plain RAG backs 45% of its claims with a quote; Ripple backs 100%, with precision recovered to 89%.
- We caught our own regression (v2's precision drop), explained it and fixed it, proven with significance tests, not anecdotes.
- Honest coverage numbers, broken down into why each miss happened.
What we learned
- A stronger model doesn't fix grounding by itself. The structure of the evidence (a graph of quoted links) does.
- "Correct" needs a precise definition. Once "partner" was pinned down, precision recovered.
- Measure against something you didn't build, and say plainly where that independence breaks down.
What's next for Ripple (not the crypto)
- Expand the name list from the reference misses (MediaTek, GlobalFoundries, TI…) and re-measure coverage.
- Add 10-Qs and 8-Ks for fresher links, revenue-share weights where filings state them, and scale to all ~4,000 US filers.
- Finish the human audit so precision has a non-model ground truth.
- Get PRISM Root Cause clustering and evaluator runs working with the PRISM team.
Log in or sign up for Devpost to join the conversation.