Did it really?

Inspiration

I run a few coding agents at once, and they love to finish with "Done! All tests pass, I didn't touch anything else." Sometimes that's true. Sometimes the tests never ran, or a file changed that nobody mentioned. Then I found a post on r/AI_Agents: "The agent exited cleanly, did nothing, and reported success."

The usual fix is asking another agent to review everything, which means writing a long prompt and explaining the whole history again. I wanted something faster: paste the report, point at the repo, get an answer.

What it does

Did it really? is a small local web app. You paste the agent's final message, point it at your project folder and pick the session (Auto, last hour, today...). It reads what actually changed in git, splits the report into separate claims and gives each one a verdict:

✅ Verified, with the exact diff lines as proof
❌ Not found, the diff doesn't show it
⚠️ Unverifiable, things like "all tests pass" that code alone can't prove

It also lists every changed file the agent never mentioned. That's usually where the surprises are.

How I built it

I used the Devpost Learn skill pack in Claude Code and went plan-first: scope.md, prd.md and spec.md before any code. They're in the repo under devpost/.

The app is Python + Flask with a plain HTML/CSS/JS page. Changes come from regular git status, git log and git diff calls. Gemini (free tier) gets the report and the diff in one request and returns structured JSON.

The part I like most: the checker can lie too. So Gemini has to quote lines from the diff, and Python checks that every quote really exists there. A ✅ without real evidence gets downgraded to ⚠️. Claims like "I didn't touch config.py" are judged by Python on its own, straight from the diff. 88 tests cover the git logic and these checks without calling the AI.

Challenges I ran into

New files. For a brand-new file Gemini quoted diff headers (diff --git, new file mode) as evidence, so true claims came out ⚠️. Now the model never sees those headers, and new files use their real added lines as proof.
A lie hidden behind a denial. In my own test repo the agent quietly changed a discount from 0 to 50 and wrote "I left config.py untouched". Because the file was named in the report, it didn't show up as unmentioned. Now denials are checked per file, and a file named only in a denial still counts as unmentioned.
Lines glued with ^. Sometimes Gemini returned evidence lines joined with ^ instead of newlines, so real proof didn't match the diff. I caught it while recording the demo video.
Windows encoding. Diffs with Cyrillic or emoji crashed on cp1251 until every git call was decoded as UTF-8.

What I learned

Planning first changed what I built. The scope made me cut test running, GitHub integration and sub-agents, and that's the reason it's actually finished. And every bug I found on real repos went back in as a test first, then a fix.