Chainwatch

An AI Agent That Proves the Bug Are Still Live On-Chain

Inspiration

I've spent about two years learning bug bounty and penetration testing — and the one pattern that never stops repeating is this: the exploit almost never comes from code that's obviously broken today. It comes from a commit, months or years earlier, where someone quietly removed a check nobody thought twice about.

I've also been deep into vibe coding and agentic engineering — building systems where an AI agent actually does real work inside a tight, verifiable loop, not just answers questions. I kept circling back to the same question: what would the smartest possible AI security agent actually look like — one that a real bug hunter could trust, not just a demo that finds something once and calls it a day?

Chainwatch is my answer to both of those obsessions at once: a security tool built around git history instead of a code snapshot, with an AI layer that does real analytical work but is never allowed to be the one deciding whether a vulnerability is real.

What it does

Chainwatch walks a Solidity repository's entire commit history, comparing every commit pair against its own real, reconstructed dependency environment — not source text alone. Ten deterministic rule classes, each mapped to an OWASP Smart Contract category, catch the moment a specific protection (access control, reentrancy guard, initializer safety, overflow checks, and more) was weakened or removed.

Finding where something broke is only half the job. Chainwatch then checks whether that exact broken version is still alive on-chain — comparing normalized runtime bytecode against real deployed contracts, byte for byte. We proved this matters on a real, historical case: a 2021 exploit ($6.5M, publicly disclosed) that was patched in source code six weeks later — but the already-deployed proxy clones running the broken implementation never got that patch, because a source fix can't reach bytecode that's already live. We verified, with a real read-only eth_call against real mainnet contracts, that the vulnerable function is still callable today.

A Gemini-powered agent layer sits on top: it drafts the disclosure report, explains the mechanism in plain language, and — critically — mechanically re-verifies every claim in its own draft against the raw evidence before anything ships. It never gets to decide whether a finding is real. That call belongs entirely to a deterministic verdict model requiring six independent evidence fields before anything reaches CONFIRMED.

How I built it

The core engine (src/scan.py) walks commit pairs and compiles both sides with Slither, using each commit's own real dependency resolution. On top of that sits a newer invariant-discovery layer that reasons about a protocol's structural guarantees (value conservation, share-price consistency, entitlement limits) instead of matching fixed code patterns — which is how it found real, previously undocumented bugs on a completely different live protocol during testing (details withheld pending responsible disclosure — more on that below).

The AI layer runs on Google's Agent Development Kit and the Gemini API, deployed on Cloud Run, with a hard architectural boundary: the model has exactly one tool (a read-only eth_call), its structured output is stripped to known fields, and a verdict is something only deterministic code is allowed to set. A self-verification pass checks every sentence of the model's own report against the raw evidence bundle before the report is considered final.

Challenges I ran into

The hardest, most persistent challenge across this whole project was getting the false-positive rate to zero without gutting recall. Every time I fixed a real gap in a detection rule, I'd sweep it against real, production Solidity from Aave, Compound, Uniswap, and others — and almost every time, that sweep surfaced a NEW false positive I hadn't anticipated: a correctly-guarded initialize() flagged as unguarded because the guard didn't use a named flag; a Maker-style oracle staleness check missed because it didn't match Chainlink's specific pattern; an upgradeable counter library missed because of a naming detail. Each fix meant going back to real code, not synthetic fixtures, and proving the fix generalized instead of just patching the one case I'd found. That discipline — measure against real protocols, never trust a synthetic fixture alone — is the single thing I'm proudest of in how this was built.

The second real challenge was realizing that "the deployed contract matches current git HEAD" is a false assumption for immutable proxy clones — a bug in my own liveness logic that, once fixed, is what let the 2021 finding reach a genuine CONFIRMED verdict end-to-end, through the unmodified CLI, with no manual workaround.

What I learned

The most important thing I learned is a discipline, not a technique: a security tool's credibility rests entirely on what it refuses to claim, not just on what it finds. Building the verdict model to require six independent evidence fields — and structurally preventing the AI layer from ever setting that verdict itself — made the tool slower to reach "yes," but made every "yes" it does reach worth trusting. In a field full of tools claiming AI can just spot vulnerabilities, I think the more honest, more useful architecture is the boring one: deterministic detection, with AI doing what it's actually good at.

I'll also say plainly: during testing, Chainwatch's invariant-discovery engine found additional critical-severity findings on a separate real, live protocol — with real user funds involved. We are not publishing what they are. A tool that discovers a live, unpatched bug and broadcasts it before the affected team can respond isn't doing security research, it's handing out a blueprint. Those findings are going through private, responsible disclosure instead. I think that restraint is as much a part of what this project demonstrates as the detection engine itself.