HypeShield

HypeShield checks whether an investment memo's headline claims are supported by its disclosed fees and cash waterfall.

Comment

HypeShield: Footnote-First Deal Audit

Paste-ready submission text for Money Talks 2 (AI × Finance), Nomad Works NYC, 2026-10-17. Deadline 4:00 PM ET.

Project name

HypeShield: Footnote-First Deal Audit

Track

Money Intelligence

Team

[TODO: Bobby Ren, Ricky Wang]

Tagline (one line)

An AI deal auditor that checks whether a private-placement pitch survives its own footnotes.

Short description (≈280 characters)

Private deals sell the headline. HypeShield retrieves the memo from a PRISM Knowledge Base, audits it through Nicrron at temperature 0, and traces every run in PRISM. It shows that a 31.5% headline IRR doesn't survive the memo's own footnotes.

Problem

Investment memos lead with the headline return, but the investor's actual economics sit in the footnotes. Our synthetic private-placement memo (Casa Azul, PPM-AZUL-2026-Q4) pitches a 31.5% IRR. Its own footnotes say:

  • An 18% gross-revenue royalty ($3.6M, Footnote 9) and $1.9M of overhead and debt service cut a $6.0M pre-royalty operating surplus to $500K of Year 3 cash for Series A investors (Footnote 14).
  • A 12% / $600K structuring fee comes off the $5M raise at closing (Footnote 11).
  • The memo itself states a 4.1% net IRR. That figure is the promoter's own and is unverified. The excerpt doesn't contain enough data to recalculate it, and we didn't try.

An LLM summarizer that repeats the headline, or skips a fee footnote, misleads the reader.

What it does

HypeShield answers five fixed audit questions about the memo. A good answer must:

  • cite the document ID plus the section or footnote for every number,
  • separate the 31.5% headline from the memo-stated 4.1% net IRR,
  • trace the waterfall from $6.0M to $500K without double-deducting,
  • flag the $600K closing fee separately,
  • say plainly that the IRR can't be recalculated from the excerpt, and give no investment advice.

We measure where a baseline prompt (V1) fails, change only the system prompt (V2), and compare the two on identical inputs.

How it's built

  • Retrieval: PRISM Knowledge Base search (Casa_Azul_Synthetic_KB.md, assigned to agent hypeshield).
  • Model: called through Nicrron: anthropic/claude-haiku-4-5, temperature 0, fixed token limit. 57 calls so far.
  • Tracing: one PRISM trace per run, with two steps: the retrieval step (carrying the actual KB text) and the LLM step.
  • Controls: the retrieved context is frozen by digest, so V1 and V2 share model, temperature, inputs and context. Only prompts/v1_system.txt and prompts/v2_system.txt differ.
  • App: Python CLI (preflight, smoke, baseline, replay, pairs, record-eval) plus a one-screen Streamlit V1-vs-V2 comparison.
  • Tests: 15 offline tests that use fake adapters and make no network calls.

The experiment

  • Baseline: 50 V1 runs = 5 questions × 10 repeats. At temperature 0 the outputs were identical within each question, so these are 5 distinct answers repeated, not 50 independent cases.
  • Replay: 5 V2 runs on the identical 5 inputs and frozen context. Only the system prompt changed.
  • Total: 56 traces, all acknowledged by PRISM (1 smoke + 50 V1 + 5 V2). Experiment exp-1017-1427-f515.

Observed results

These come from our own local text checks (a regex checklist). They are not PRISM evaluator scores.

Question V1 baseline gap (identical across all 10 repeats) V2
All 5 Never cited the document ID PPM-AZUL-2026-Q4 in any of the 50 runs Fixed
Q1: audit the 31.5% IRR Got the waterfall right but omitted the 12% / $600K closing fee (Footnote 11) Fixed
Q3 No footnote citations at all Citations added, but it regressed by dropping the $600K fee
Q4 Never said the IRR can't be recalculated from the excerpt Fixed
Q5 Dropped the $1.9M overhead line from the waterfall Fixed

Takeaway: V1 had one blind spot, a different missing footnote in each question. V2 fixed most of the gaps but introduced a regression on Q3, which is why every fix needs an evaluator check.

PRISM proofs (honest status)

  1. Knowledge Base retrieval in a run: DONE. The KB document is assigned to hypeshield. Smoke trace 44c078e6-d44d-4d7c-b47d-f1ad13e420dc shows hypeshield_audit → prism_kb_search (with the retrieved memo text) → the LLM call.
  2. Root Cause cluster of 3+ failed runs: PENDING. It's blocked by the judge-model error described under Challenges. Our own hypothesis, which is not a PRISM label: V1 is told to cite "relevant source sections" but not to walk every fee footnote, so it drops the separate Footnote 11 fee and the document-ID citation.
  3. Custom evaluator, V1 vs V2 on identical inputs: PENDING, for the same reason. The evaluator, Footnote-First Grounding Score (0–100, pass at 80, capped when citations are missing), is written. The 10 traces it will score are fixed in advance: V1 repetition 1 plus V2 for each question. No PRISM scores exist yet, so none are reported here.

Challenges

  • PRISM judge model returns 401. Every trace's automatic analysis reads evaluation_failed: 401 from api.anthropic.com: invalid x-api-key. That blocked automatic analysis, the Evaluator and Root Cause, so we are reporting those proofs as pending rather than simulating them.
  • Temperature 0 makes repeats identical. That gives us a reproducible baseline but not statistical spread. We report it as 5 × 10 rather than 50 cases.
  • A bug in our own fixture. The first draft of the memo called $6M "EBITDA" while also deducting the royalty before it, a double deduction. We redefined $6M as pre-royalty operating surplus and added a startup check that rejects the old wording.

What's next

  • Once PRISM's judge works: run the evaluator on the 10 pre-chosen traces and run Root Cause on the V1 failures.
  • Write a V3 prompt that fixes the Q3 regression, and gate every prompt change on the evaluator score.
  • Extend from one synthetic memo to a set of public offering documents, with a footnote checklist per deal type.

Links

Built With

  • anthropic
  • block-convey
  • claude-haiku-4.5
  • httpx
  • nicrron
  • prism
  • python
  • streamlit
Share this project:

Updates

Submission history