Stale Hunter

Inspiration

Wikipedia is the textbook every AI learns from, and the first source search engines and chatbots repeat. But facts drift: CEOs change, rankings move, planned launches slip or get cancelled. Wikipedia itself knows this. Over 45,000 articles are officially tagged "in need of updating" and 87,000+ contain statements marked "potentially dated", some of them for more than a decade. Volunteers flag stale facts by hand, and no approved bot checks whether a fact is still true. We wanted an agent that does the tedious part: find the stale facts, prove they're stale, and hand an editor a ready fix.

What it does

Paste any Wikipedia article. Stale Hunter:

1. Scans every sentence and flags the ones that can go out of date: office holders, rankings, "as of" figures, statuses, plans with past dates.
2. Dates each candidate from Wikipedia's edit history (via WikiWho), so sentences nobody has touched in years are checked first.
3. Researches each fact on the live web with Tavily, skipping Wikipedia and its mirrors.
4. Judges each fact with a reasoning model that must separate "the world changed" from "sources disagree", and digs deeper with a sharper search when the evidence is thin.
5. Verifies: a second, different Nemotron model must independently agree before anything is marked outdated.
6. Fixes: every confirmed issue gets a rewritten sentence plus {{cite web}} references, ready to paste into the Wikipedia editor, along with when the sentence was written and how long it has sat untouched.
7. Finds what's missing: in parallel, it reads the past year's news about the subject and surfaces notable developments the article never mentions (an acquisition, a new CEO, a closure), each with a draft sentence and the section it belongs in, also confirmed by a second model.

Any language: paste a link from any Wikipedia edition. Smaller editions have far fewer editors, so facts drift faster. On Hindi Wikipedia, the opening line of Virat Kohli's article still calls him captain of all three formats; Stale Hunter flags it, names the current captains, and writes the fix in Hindi.

Beyond Wikipedia:

• Any web page: paste a docs page, help centre or blog URL. It's read with Tavily Extract and checked the same way, with the page's own site excluded from the evidence.
• An API for AI agents: POST /api/check {"statement": "…"} returns whether a fact is still true, the current fact and sources in about 10 seconds, so an agent can check before it repeats something.
• Editor tools: copy all fixes as wikitext, a ready Talk-page note, an edit summary and a share link.

You watch it happen: the article scrolls under a scan line, the reader follows each verdict as it lands, and the result collapses to an A–F freshness grade with only the flagged passages.

Real finds include Manchester United "ranked third" in the Deloitte Money League (8th in 2026; that sentence sat untouched 16 years), ISRO's Venus mission "scheduled for 2025" (now 2028), and a Kenyan carmaker whose page still says production "will commence as soon as capital is raised" six years after it went into liquidation, and never mentions its 2025 rescue by new owners. On Infosys, it surfaced the July 2026 appointment of a CEO-designate the article didn't yet mention.

How we built it

• NVIDIA Nemotron on Nebius Token Factory, each model doing the job it's best at:
  • Nemotron 3 Super with thinking off scans the article in parallel chunks: fast and cheap, about 2 s per chunk.
  • Nemotron 3 Ultra with full reasoning judges each fact against the evidence.
  • Nemotron 3 Super with reasoning gives an independent second opinion on every red flag.
  • The UI shows every model call, its latency and its tokens ("How Nemotron did the work").
• Tavily: advanced search for evidence, news search (past year) plus structural-change queries for missing developments, basic search for the dig-deeper loop, and Tavily Extract to read any web page.
• WikiWho (Wikimedia Cloud) for token-level edit history, which drives both prioritisation and "untouched for N years".
• Next.js 15 on Vercel, streaming NDJSON events to a live UI.
• Guardrails: sentence-ID referencing (the model can't misquote the article), prompt-injection-aware prompts, verdict rules enforced in code, retry/backoff for rate limits, a result cache and per-visitor limits on the public demo.

Challenges we ran into

Our first version was confidently wrong: only about 2–3 of 10 "outdated" verdicts were real. It confused historical corrections ("that happened in 2014, not 2015"), rounding and different definitions (metro vs. city GDP) with staleness. We built an evaluation harness (29 articles, hand-reviewed verdicts) and iterated: a change-type classification, a past-anchored-statement rule, a second-model verifier and a dig-deeper loop. On the final run, about 8–9 of 11 outdated verdicts held up in hand review, unverifiable results dropped from 106 to 47, and model-call failures went to 0 of 340.

A subtler trap: stale sources "confirming" stale facts. Old, undated profile pages still say Kohli is captain, and the judge once called that statement current, even inventing dates for the sources. Now the judge may never assume a source's date, and only recent evidence can confirm that something is still true.

We also hit Wikipedia rate limits (fixed with a Wikimedia-compliant User-Agent and backoff) and run-to-run variance (fixed by ranking candidates on edit-history age, which raised sentence overlap between repeat runs from about 0.5 to 0.67–0.82).

Accomplishments we're proud of

• A Hall of Stale: 16 outdated facts across 32 audited articles, each confirmed by two models and reviewed by a human, sitting untouched for 5.7 years on average.
• Turning a 2025 research idea (WiNELL, Wikipedia Never-Ending Updating with LLM Agents) into a usable tool.

What we learned

Fact-checking is mostly about what not to flag. The hard part wasn't finding new information; it was teaching the agent the difference between "changed", "wrong" and "described differently", and making two models agree before raising an alarm.

What's next

• A bulk backlog run over Wikipedia's 45,000 "needs updating" articles with Nebius Serverless Jobs
• Bulk runs across smaller Wikipedias (Hindi, Marathi, Tamil…), which have the fewest editors per article
• A browser extension and watchlists for editors and docs teams, re-checking pages on a schedule
• An MCP server wrapping /api/check, so any agent framework can call it as a tool

Built With

• mediawiki-api
• nebius-token-factory
• next.js
• nvidia-nemotron
• tailwindcss
• tavily
• typescript
• wikiwho