Landing page
Outcome

Sherlock

Inspiration

Sherlock started with a currency puzzle I could not solve.

For months I watched the euro strengthen against the dollar while losing ground against the pound. The rates were internally consistent; the explanation was not. Every visible narrative — interest rates, macro data, headlines — failed to account for the pattern. The only conclusion left was that the decisive information was something I could not see.

That reframed the problem for me.

Sherlock Holmes solved Silver Blaze because of a dog that didn't bark. Physicians perform differential diagnosis by ruling hypotheses out. Popper argued that science advances by falsifying hypotheses rather than confirming them. Across very different disciplines, breakthroughs often come from identifying what should have happened , and didn't. Then I noticed the same blind spot in the AI tools I was using. They are excellent at explaining the evidence in front of them, but they rarely look for the evidence that should exist and doesn't. Once a model finds a plausible explanation, it naturally tends to reinforce it instead of actively testing competing ones. That is useful if your goal is to answer questions.

It is dangerous if your goal is to investigate.

What happens if we force an LLM to think like an investigator instead of an assistant?

Sherlock is my attempt to answer that question.

It is not another prompting technique.

It is an investigation framework. The prompt is only one component of a reasoning workflow that constrains how the model is allowed to investigate a problem.

The goal is not to generate a better answer.

The goal is to make the model earn the answer.

What it does

You give Sherlock four things:

What happened.
What should have happened (normal behavior).
The available evidence.
Optionally, your own hypotheses.

Instead of returning a narrative, Sherlock builds a structured investigation.

It produces:

An Expectation Matrix. Every observation is classified as expected or unexpected, present or absent. The expected-but-absent quadrant receives special attention because missing events often discriminate between competing explanations better than observed ones.
Competing hypotheses. Sherlock keeps multiple explanations alive simultaneously. Every hypothesis must explain what supports it, what contradicts it, and exactly what evidence would falsify it.
A Hypothesis Graveyard. Rejected explanations are never deleted. They remain attached to the precise conditions that would make them plausible again.
Missing evidence ranked by value. Sherlock identifies which observations would reduce uncertainty the most instead of asking for more information indiscriminately.
The next test. Rather than pretending certainty, Sherlock recommends the single observation with the highest power to distinguish between the leading hypotheses.
Iterative learning. New evidence updates the existing investigation instead of starting over, preserving confidence changes, hypothesis transitions and reasoning history.

The demo case is one many engineers recognize.

A checkout starts returning HTTP 500 errors shortly after a deployment. Everyone blames the deployment and rolls it back.

The errors continue.

Sherlock weakens the deployment hypothesis and instead surfaces something nobody looked for: the TLS certificate renewal log that should have appeared at 23:05 but never did.

The engine is intentionally domain-independent. Nothing in Sherlock is specific to software incidents.

How we built it

This project was built by a product manager who does not write code.

I designed the investigation methodology, the reasoning framework, the product, and the technical specifications.

OpenAI Codex implemented them.

Development was organized into disciplined implementation blocks, each driven by a written brief with explicit acceptance criteria and one governing rule:

One finished block beats three half-started ones. Unfinished work is finished or reverted, and the repository stays green.

The project evolved through four major stages:

Block 1 — Contract. A single canonical investigation schema became the source of truth for every component.
Block 2 — Investigation engine. The falsification workflow and semantic evaluation harness were implemented.
Block 3 — User interface. The complete investigation experience was built directly on the canonical schema.
Block 4 — Iterative reasoning. Sherlock learned to continue investigations as new evidence arrived instead of restarting them.

The reasoning engine runs on GPT-5.6 Structured Outputs.

The model is not used as a conversational assistant. It follows a versioned investigation workflow with explicit rules for hypothesis generation, falsification, evidence management, confidence updates and iteration.
Every response is validated against the canonical JSON Schema using AJV before reaching the application.
The result is an investigation that can be inspected, validated, replayed and audited.

Challenges we ran into

The hardest challenge was making the model reason about missing evidence without inventing it.
An absence only counts if it derives from the declared expected behavior and could actually have been observed with the available instrumentation. -The silence of a sensor that never existed proves nothing. -Another challenge was distinguishing between a hypothesis that is no longer useful and one that has actually been falsified.
Sherlock now treats those as different states through an explicit hypothesis lifecycle.
Perhaps the most interesting challenge appeared during evaluation. - One semantic test failed, suggesting Sherlock had chosen a poor next investigation step.- Instead of immediately changing the prompt, we inspected the investigation artifact itself.- The reasoning was correct.- The bug was in the evaluator.

We restored the original investigation prompt unchanged, rewrote the evaluator to measure structural discrimination instead of prose heuristics, added regression tests, and the next evaluation passed every semantic assertion.

That became one of the project's guiding principles:

Inspect the evidence before changing the instrument.

Accomplishments that we're proud of

Building an end-to-end AI product without writing application code by hand, using OpenAI Codex as the implementation partner.
Designing an investigation framework instead of another conversational workflow.
Creating a semantic evaluation harness that validates reasoning rather than only schema conformance.
Building iterative investigations where confidence evolves instead of being recalculated from scratch.
Delivering a complete product with a working demo, offline investigation replay, strict validation and an auditable reasoning process.

What we learned

We started believing that better AI systems require better models.
We finished believing they also require better reasoning frameworks.
Large language models naturally optimize for plausible explanations.
Investigation requires something different: competing hypotheses, explicit falsification, disciplined evidence management and continuous revision as new evidence appears.
We also learned that AI-assisted software development depends as much on the quality of the specifications as on the capability of the model implementing them.
Codex did not simply accelerate development. It made development delegable. That allowed the human contributor to focus on product design, reasoning methodology and judgment.