Doublecheck

Agents say done. Who double-checks? Doublecheck catches a valve reported closed but leaking, a geyser slowing over weeks though every cycle ended fine, an odd cleaner run, else Not verified if unsure.

Comment

Inspiration

Agents and smart-home routines now act for us while we're away, and they report back with one word: done. But "done" is a report, not proof. A valve can say it is closed while water is still flowing. A geyser can finish every cycle and still take twice as long as it used to. A robot cleaner can "complete" a job after stopping five times. The assistant sees the command succeed while the real outcome is something else.

Doublecheck isn't another smart-home assistant. It's a verification layer for agents that act on your behalf.

What it does

Doublecheck is a self-hosted MCP server (with a simulated Alexa+ screen on top). It follows one chain: what was requested → what should have happened → independent evidence → historical behaviour. It compares what a device or agent reported with independent evidence and its own history, and answers one of three things: Worked, Problem found, or Not verified. Every answer shows its evidence, the time window it used, and what it cannot tell you.

Three scenarios show three classes of agent failure:

Water valve (contradictory outcome). It reports closed, but an independent flow meter still reads about 6 L/min after a 90-second settling window. Doublecheck says Problem found. Approve a simulated re-send of the close command and it checks again afterwards; only then does it say Worked.

Bathroom geyser (degradation over time). Every cycle ended fine, yet recent cycles take about 20 minutes where comparable earlier cycles took about 12.5, roughly 1.7 times as long, built up over five weeks. Doublecheck compares only cycles with a similar temperature rise, flags the sustained change, and never states a cause.

Floor cleaner (failure hidden inside success). It completed, but with 5 pauses (usual 0) and 63.8 minutes (usual 48.7), judged only from the device's own run log.

A home report ("Was everything actually done?") ties these together and keeps timing honest: right now, earlier today, from the last run, over recent weeks, could not confirm, confirmed. You can open the report for any of the last 30 days, and it uses only what had been recorded by then.

Why "Not verified" matters.

A front-door lock with no independent sensor stays Not verified. Doublecheck does not guess, and it does not call that a failure either. When evidence is insufficient, it refuses to manufacture certainty. A system that says "I can't confirm this" is more trustworthy than one that confidently reports success.

How we built it

A deterministic TypeScript verification engine with no language model involved in verification decisions. The same inputs produce the same result, making findings reproducible and testable.

A device-agnostic design: each device is described by its expected outcome, the independent evidence that can confirm it, and its history, without needing a separate verification engine.

A hand-written MCP server over Streamable HTTP (specification version 2025-11-25), with 11 tools, structured results, and an approval step before any fix is applied.

An append-only local store that preserves past findings, and a React interface that communicates with the server over MCP.

A fault-injection benchmark that measures false reassurance (a broken device incorrectly treated as fine) rather than accuracy alone, alongside automated tests across the verification engine and MCP layer.

Kiro Crew contributed a focused improvement to recurring-problem reporting: Doublecheck now surfaces earlier problems for the same device from the preceding 30 days.

Challenges we ran into

Avoiding false certainty. A longer heating time is a warning sign, not proof of a fault. Doublecheck flags the geyser's changing behaviour without pretending to know its cause.

Accounting for sensor failure. A flow sensor stuck at zero looks exactly like a closed valve. Our benchmark shows 15.5% false reassurance with one sensor; with two sensors, the outcome becomes Not verified when the evidence cannot establish the truth.

Keeping time context honest. A problem observed last week must not be presented as happening now. Findings keep their time scope, so historical trends, completed-run anomalies and current problems are never confused.

Designing for a person who isn't watching. The home screen puts findings that need attention first and keeps routine device status quiet, so exceptions are visible without overwhelming the user.

Accomplishments that we're proud of

A working verification layer, not just a concept. A deterministic engine runs through a self-hosted MCP server that we built by hand, connected to a working React interface.

Three ways "done" can be misleading, caught by one engine. A valve reports closed while flow continues, a geyser's heating time degrades across weeks while every cycle ends fine, and a cleaner completes its run despite abnormal pauses.

Uncertainty as a first-class outcome. Doublecheck distinguishes Worked, Problem found and Not verified, so a device with no independent evidence, like the front door lock, is never guessed at.

Honest numbers. Our benchmark reports false reassurance, not just accuracy, and shows 15.5% with one sensor stuck at zero. We published the weak case instead of hiding it.

A report that respects time. You can look at any of the last 30 days, and each day uses only what had been recorded by then. Past problems are never presented as current ones.

Safe fixes. An approval step comes before any fix, and Doublecheck re-checks afterwards and only then says Worked.

Honest about its limits. It does not diagnose causes, and it does not pretend synthetic readings are real-world evidence. It shows what the available evidence supports, and what it doesn't.

What we learned

A successful command is not the same as a successful outcome. Reliable verification requires defining what should have happened, checking what the evidence actually shows, and comparing it with historical behaviour. Just as importantly, it must recognize when the available evidence is insufficient.

Sometimes the most trustworthy answer isn't "Done" or "Failed." It's "Not verified."

What's next for Doublecheck

Connect real sensor feeds. Integrate flow meters, smart plugs and other independent sensors so verification can move beyond synthetic data and into real-world outcomes.

Expand devices and histories. Support more device types and longer histories to spot gradual changes with more confidence. Because each device is described by its expected outcome, confirming evidence and history, new devices can reuse the same verification engine.

Let agents verify before they say "done". Enable agents to call Doublecheck after an action and before reporting completion, so success is supported by evidence rather than assumed.

Validate in real homes. Test against real devices and Alexa+ routines, measure what the engine misses and incorrectly flags, and improve its handling of conflicting or unreliable sensors.

The long-term vision is a reusable verification layer for agentic systems. It is not another assistant that issues commands, but one that checks whether those commands achieved their intended outcomes.

Alexa can say it is done. Doublecheck asks whether the evidence agrees.

Built With

Share this project:

Updates

Submission history