Inspiration
I kept making plans or goals only to later realize that they wont work out anymore.. just because with the rise of AI the world is moving too fast. The company I was aiming at started laying people off. An idea I thought was mine had already been shipped. A deadline I had built a revision schedule around had quietly changed.
Nothing was wrong with any of those plans when I made them. They just went stale, and nobody told me.
The assistants I used could not help with this, because they are all reactive. They answer when asked and forget afterwards. The problem is not that I cannot look something up — it is that I do not know when to go and look again. You cannot search for news you do not know exists and especially since the volume of news have increased so drastically.
So I wanted something that would hold on to what I am planning and come back to me when the ground underneath it shifted.
What it does
You tell The Falsifier a plan or a belief. It then:
- Works out who actually decides. For a claim about an exam, it identifies the examining board and verifies that domain resolves before trusting it. Users do not name their sources; working that out is the agent's job.
- Breaks the claim into what it rests on, and says which parts are too vague to check.
- Invents concrete ways each part could be false — before searching anything. This is the core of it. A search engine returns what you asked for, so asking "is my plan fine?" returns reassurance. You have to work out what the counter-evidence would look like first, then hunt for that.
- Searches for the counter-case, grades every source by who published it, and reads the survivors.
- Returns one of three honest answers, all cited: this still holds, this no longer holds, or I cannot settle it — and here is exactly what would.
Every finding carries the exact sentence it rests on and a live link.
Then it keeps watching. Each cited source is re-checked on its own schedule — three days for an official page. When a quoted line disappears from the page it came from, the finding is withdrawn, with the date and the reason, and the conversation it belongs to lights up with a message that arrived without anyone asking.
That last part is the whole product. It looks like any chat app until a thread you have not opened in a week is sitting there with something you needed to know.
How we built it
Python, FastAPI, SQLite. Three NVIDIA Nemotron 3 tiers on Nebius Token Factory, Tavily for retrieval.
The architecture is the tiering. Per claim: 462 Nano, 26 Super, 2 Ultra.
- Nano invents the ways a claim could be wrong, writes the search queries, and triages hundreds of results for relevance. Short, cheap, high-volume.
- Super reads the surviving pages and extracts the exact quoted line.
- Ultra decides what actually holds, and reasons past the citations where the sources run out.
Thousands of cheap judgements to produce three sentences worth reading — only affordable because open models make volume cheap. Measured from logged calls: $0.095 per claim, against $0.899 to route the same work entirely through Ultra. A 9.4× saving.
Two rules are enforced in code rather than in prompts, because a language model will produce a fluent claim about anything:
- An extracted quote must appear word for word in the retrieved page. A string comparison, after normalising whitespace and smart quotes. About a third of attempted quotes are discarded by it.
- A verdict either cites a source or explains why it cannot. That is a
CHECKconstraint in the database, so an unsupported conclusion is unstorable rather than discouraged.
Challenges we ran into
A silent failure that cost a day. The Nemotron 3 tiers are reasoning
models, and thinking tokens bill against max_tokens. Run out and you get
an empty string with finish_reason="stop" — no error, and
indistinguishable from the model choosing to say nothing. It happened 9
times in 513 calls before I built a guard that refuses any call below a
token floor.
It said "I cannot check this" to three claims in four. The cause turned out to be a number: the confidence ceiling for a weak source sat below the floor for taking any position at all, so anything resting on a blog was automatically unsettled. It also needed a stage that reasons past the citations — clearly labelled as reasoning — because most useful questions have no single page that settles them.
It was unusably slow. A run took 220–334 seconds until I measured where the time went and found every triage and read call queued behind the last for no reason. Running them eight at a time brought it to 58 seconds with identical output.
It once withdrew a finding that was perfectly fine. The source was a government PDF; my code strips HTML tags, which turns a PDF into garbage, so the quoted line looked "missing". A false retraction is the worst failure this product can have — it tells you something collapsed when nothing did, and a tool that cries wolf gets muted. It now refuses to withdraw anything it could not actually read.
Accomplishments that we're proud of
The guarantees are in code, not in prompts. The quote check is a string comparison. The citation requirement is a database constraint. Neither can be talked around by a model having a bad day, and the quote check throws away about a third of what the model tries to use.
It is willing to disappoint you. It agrees when you are right, says it cannot tell when it cannot, and names the one fact that would settle it. An assistant that answers everything is not one worth trusting.
It publishes its own worst numbers. I expected the tier split to save around 20×. Measured, it is 9.4×. The README says 9.4×, because a judge can recompute it from the same logs and a flattering number would be worse than a modest true one.
It writes its own methods. When a claim does not fit a known class,
Super writes a new falsification routine for that domain — its angles, its
authoritative sources, its search vocabulary — and stores it. A claim about
becoming a ballet dancer produced a routine the model named
late-entry-profession, which later matched a claim about taking up violin
at thirty.
207 tests, none of which need a network or cost anything.
What we learned
A month ago I had never built an application.
The thing I did not expect was how much of the work was deciding what the tool is not allowed to say. Most of the engineering here is refusals: no claim without a verifiable quote, no confidence a weak source cannot bear, no retraction without having read the page, no answer where "I do not know" is the truth.
The other lesson was that I could not reason my way to any of this. Every significant improvement came from running it on something I actually cared about and watching it fail. The fabricated citations, the three-in-four refusals, the false retraction, the slowness — none of those showed up in tests. They showed up when I asked it about my own life.
What's next for The Falsifier
Making the plan, not just checking it.
Right now it tells you your plan has gone stale. It should help you build the next one — finding the resources, courses and opportunities you would not have known to look for, and bringing them to you. The hard part is not retrieval; it is knowing enough about your situation to tell an opportunity from noise.
Nearer term: longer-lived memory of the situation behind each plan, so it sends you fewer and better things, and watching more source types than plain web pages.
Log in or sign up for Devpost to join the conversation.