Needle in the Haystack

Inspiration

Every pharma 10-K, earnings call and FDA label is public. The case told us where the findings hide anyway: in risk-factor language that drifts a few words a year, in inventory that grows faster than sales, in a footnote reserve that doubles quietly, in the metric a CEO stops mentioning. No analyst can read ten companies' filings side by side at that resolution. We wanted an analyst that could, with one rule it can never break: every sentence it writes must point to the page it came from.

What it does

Needle in the Haystack is a cited research engine over a verified evidence base, with a two-stage decision pipeline on top.

Ask it about any of the ten drugmakers in the case (or any company or drug you onboard) and it walks the case's own list of hiding places: risk-factor and MD&A drift between annual reports, gross-to-net, inventory and receivables versus revenue by quarter, segment restatements, footnote extracts, call Q&A with deflection and new-phrase flags, side-by-side label sections against the rivals the case lists, and IRA, MFN and tariff exposure. Every row it returns carries a page citation you can click and a verification status. When no public source exists, it says so explicitly instead of guessing.

Part 1 ranks the companies blind and writes the five findings, the sharpest needle and the product fine print. Part 2 makes one CEO call per company with budget, return math, two rejected alternatives, kill signals, decider evidence and a promotional claims map that only allows a comparative claim where a head-to-head trial exists.

How we built it

The evidence base. 7,005 documents, 78,392 pages and 118,329 fact rows: annual and quarterly reports back to FY2021 for the shortlist and their rivals, ten quarters of earnings calls with quotability grades, every current FDA label split into citable sections, CMS Part D and Part B spending, Medicare negotiation selections, ICER reports, 1,600 deals with terms quoted verbatim from the acquirers' 8-Ks, patent and exclusivity tables, press-attention series, and the case's policy forces quoted cell by cell. It all sits in one SQLite index with full-text search over pages and facts.

The citation contract. Every fact carries a locator of the form doc_id#p<page> or a dataset URL, plus a verification status. A verify_quote check re-reads the page and confirms the quoted text is there, so a number cannot enter an output unless the page agrees. Transcripts are graded A (official or Bloomberg), B (two independent sources agree), C (single third party) or D (poor). A B grade requires executive-word agreement $ a \geq 0.90 $ and coverage $ c \geq 0.80 $ between the two copies.

The pipeline. Ten blind readers walk the eight retrieval helpers for every company. A materiality skeptic and an obviousness skeptic argue each candidate over two rounds, a synthesizer rules, and a ranker seals the top three with a SHA-256 hash before any writer sees it. In Part 2 a strategist proposes four or more cited moves, a CFO skeptic and a compliance skeptic try to kill each one, and a synthesizer picks one. Every figure passes the per-number citation gate.

Reusability. One command onboards a new company or drug into the same contract. Regeneron, Novartis and Vertex were built from nothing in about 190 seconds each, and the same 47 tests pass.

from agent.moat import cite

hits = cite.search_pages("transition tax notices of proposed adjustment", ticker="MRK")
citation, url = cite.cite(hits[0]["doc_id"], hits[0]["page_no"])
# "Merck 10-K FY2025, p. 129", https://www.sec.gov/...
assert cite.verify_quote(hits[0]["text"][:200], hits[0]["doc_id"], hits[0]["page_no"])

Challenges we ran into

Rate limits are data-quality bugs. Our press-mention collector overran GDELT's one-request-per-five-seconds rule, was refused, and wrote the refusals down as zero counts. A zero that means "the API said no" is worse than a blank. We re-fetched at the proper spacing and added a query_failed status so a refused query can never masquerade as an empty result.

Labels move without changing. Repackager uploads shift a label's DailyMed version date while the clinical sections stay identical, so a naive staleness check flagged fourteen drugs that had not changed. We compare section text, not dates.

Keeping the model blind. The retrieval team and the reasoning team worked on the same index at the same time without leaking findings. Only call signatures and table schemas flow to the model, and the sealed ranking carries a hash.

Paywalls. Of 71 coverage gaps against the case's own source list, we closed 62. The remaining nine are paywalled transcripts, and we say so rather than fill them from memory.

Accomplishments that we're proud of

A citation gate that holds: every number in every output resolves to a page or a dataset row, and the index contract never changed after the freeze.
An AI-error log with 514 entries, each a confident collector mistake caught by re-reading the source: a $100 billion market forecast recorded as an upfront payment, takeover-target debt ten years stale, a psoriasis source that was a tobacco biomarker paper, a prevalence figure marked verified against a paper that never mentions it.
Explicit negatives as first-class data: "no head-to-head trial exists" is recorded for 16 product pairs with the queries run, so the claims map can refuse a comparison with evidence.
Blind selection: the top three companies were picked before anyone on the team knew which they would be.

What we learned

The most valuable rows in a dataset are the negatives. "No public source states this" is a finding when you can show the query you ran, and it is the only honest way to keep an assumption labelled as one.

AI collectors are wrong in ways a human rarely is, and they are never caught by asking whether they are sure. They are caught by making them re-read the page. Verification has to be mechanical, per number, and impossible to skip.

What's next for Needle in the Haystack

Onboard the rest of large-cap pharma and the top biotech peers, add a nightly refresh that re-runs the hiding-place helpers when a new filing or label version lands and flags what moved, and open the cited API so an analyst can ask a question in plain language and get back only sentences that show their source.