Shadow CPI

AI agent that estimates India's inflation before the official release, on Nemotron (Nebius) and Tavily, with a time lock against future leaks: 9 of 20 leaked without it; 0 of 42 detected with it.

Comment

In 60 seconds

The problem. To test an AI forecaster fairly, you replay the past: "It is 29 July. What happens next?" If the web search it uses returns pages written after 29 July, the AI has already seen the answer. The test looks great and means nothing. This is called look-ahead leakage, and it quietly inflates AI backtests.

What we measured. We tested Tavily, the search API built for AI agents, on past questions. With an ordinary search, 9 of 20 questions returned at least one page dated after the cutoff with no Internet Archive copy from before it, and 14 of 22 in a second, pre-registered set (we wrote down the method and the pass rule before running it). With Tavily's date window and published-date filter switched on, we detected 0 of 42.

What we built. Shadow CPI, an AI agent that uses that time-locked search to estimate India's monthly inflation (CPI) about two weeks before the government publishes it. Each estimate is sealed with a cryptographic fingerprint, so anyone can check it was not changed after the official figure came out. First sealed forecast: October 2026, 6.09%; the official figure is due 12 Nov.

Who it is for. Anyone who tests AI with web search: forecasting researchers, economics and quant teams, and agent builders. The time lock is reusable as a Python package and as an MCP tool.

Try it in one minute

  • Open the app: it starts on the Leak demo, a real petrol-price question. The ordinary search is on the left (one result marked LEAK), the time-locked search on the right (every result marked OK). No key, no credits.
  • Try it page: ask your own question with a cutoff date; it runs live on our Tavily credits, with a small daily cap.
  • Reproduce it: the Colab notebook re-runs the leak check with your own Tavily key; python -m pytest -q runs the tests with no keys.

How it works

  1. Search (Tavily Search): every search sends start_date/end_date ending on the cutoff, with filter_by_published_date always on. Our code refuses settings that pick their own window (time_range, days, auto_parameters) and re-checks every date.
  2. Read (Tavily Extract): returns the page, which we show to the model as numbered lines.
  3. Screen (NVIDIA Nemotron Nano on Nebius Token Factory): a fast yes/no: "does this page state an Indian price or price change?" 622 calls sent, median 1.3 s.
  4. Cite (NVIDIA Nemotron Super on Nebius Token Factory): returns the line number for each price. Code copies that line from Tavily's page and checks the number is really in it. A model never writes a quote or chooses a number. 719 calls sent, median 2.3 s.
  5. Compute (plain code): prices are combined with the government's official weights.
  6. Seal: the forecast file's SHA-256 fingerprint and a git tag are published before the official release. The app's chain of custody traces each cited number back to its quote and page (and, for the later runs, the Tavily request ID).

How we use Nemotron on Nebius Token Factory

  • Two model sizes behind one OpenAI-compatible endpoint: the small model handles the volume, the large one does the careful citing. Roles are picked by model name in config.
  • Strict JSON-schema output worked on the first call, so every answer is parsed, not guessed.
  • Citing line numbers instead of copying text made Hindi pages work.
  • A content check for leaks the date filter cannot see: Nano flagged a related-stories item from after the cutoff inside a page dated before it: 1 confirmed case in 55 sampled excerpts. It also raises false alarms, so a person reviews every flag.
  • Code backstops for model slips: a page with a visible rupee price is never skipped; the item name must be in the cited line; a price range takes its midpoint.

How we use Tavily

  • Search with start_date, end_date, filter_by_published_date and include_published_date, wrapped in a time lock that cannot be switched off.
  • Extract to read every cited page; Map and Crawl to find official price lists and tariff orders.
  • include_usage on every call, so credits are logged per result and capped per run.
  • MCP server with three tools (search, as_of, check_leaks), so any agent can use the same time lock with its own key.

What makes it non-obvious

  • A search date picks which page, not which version of it. Example: Tavily dated a TCS results preview 7 Jul, before an 8 Jul cutoff, but the page was edited on 9 Jul with a link to the actual result, and Extract returns today's edited page. So we built extract_as_of: it returns the Internet Archive's copy from on or before the cutoff, or refuses. (One demonstrated case, not a rate.)
  • Should the model write its own searches? On 29 known price changes, fixed code templates found 6, Nemotron-written search fields 2 and 2, and Tavily's auto_parameters 0. So the model chooses words and sources; code owns the dates.
  • Forecasts that cannot be quietly edited: the seal is published before the government's release, and the October forecast is scored against it on 12 Nov.

Results

  • Leaks: ordinary search 9 of 20 and 14 of 22; Tavily's time lock 0 of 42 detected (95% interval 0–8%).
  • The cost of being strict: the time-locked search did not return 12 of 40 pieces of valid evidence that the ordinary search found. Correctness costs some recall, and we report it.
  • Forecast accuracy (Jan–Sep 2026 replay, pre-registered, one disclosed correction): average miss 0.194 percentage points, against 0.212 for a trend-only model and 0.468 for repeating last month; about a tie with the Reuters economists' poll on the 6 months it exists (0.150 vs 0.148).
  • What Tavily added: the sealed October estimate is 6.090% with Tavily's web evidence and 6.202% with government price series only. The 12 Nov release will show whether it helped.

Honest limits

  • Small, India-focused samples. "0 detected" is not "0 exist": the Internet Archive confirmed 64 of 114 time-locked results as published before the cutoff and none after it; the other 50 could not be checked.
  • Leak counts for ordinary search are lower bounds: we only count a leak when the archive has no copy from before the cutoff.
  • The time lock controls what search returns, not what the models already know: Nemotron Super's post-training (February 2026) falls inside the replay, so the sealed forward forecasts are the clean test.
  • The seal proves when a forecast was committed, not what was read before it; the time lock and extract_as_of cover the evidence.
  • A post-submission re-read found a finance-check result dated on the cutoff day whose search excerpt included the next day's results headline (from a related-stories box). Date checks pass it, so our "safe" label means date-safe, not content-safe; this is why we ask Tavily for as-of excerpts.

Feedback for Tavily

Conflict of interest. This hackathon is hosted by Nebius, which owns Tavily. No one on the team works for Nebius, NVIDIA or Tavily, and the measurements are reported as found.

Three feature requests, each with evidence and a one-request repro script in the repo:

  1. Say where published_date comes from and how confident it is. It was later than the page's own date on 4 of 42 date-test pages.
  2. An as-of (snapshot) mode for Extract and Search excerpts, or at least the crawl time of the copy returned.
  3. Date parameters on /research, which has none today, so research calls cannot be time-locked.

What's next

  • 12 Nov, 4 pm IST: India publishes October CPI; we score the sealed forecast against it in public.
  • About 1 Dec: the November seal.
  • Beyond the hackathon: the same pre-registered test on other search providers.
  • The judged version is the git tag submission-v1.0; anything committed later is labelled with its date.

Links

Built With

  • mcp
  • nebius-token-factory
  • nvidia-nemotron
  • python
  • streamlit
  • tavily
Share this project:

Updates

Submission history