Ratchet

Ratchet walks git history to find security regressions others miss — links every one to its cause, checks if it's still live today, and writes the evidence straight into DataHub's graph.

Comment

Ratchet

Every other tool reports the state. We report the trajectory.

Inspiration

It started with a simple, uncomfortable question: when an AI coding assistant fixes a bug, what does it actually do to the security around that bug?

We kept seeing the same pattern in commit histories — a permission error, followed minutes later by a commit that quietly loosens the exact control that caused it. USING (auth.uid() = owner_id) becomes USING (true). The error disappears. The dashboard loads. Nobody notices that the database is now readable by anyone with the public key.

We went looking for a tool that could catch this — and found none. Secret scanners look for leaked credentials. Static analyzers look at the code as it exists right now. Every one of them answers the same question: what's the state? Not one of them asks: when did protection change, why, and is it still broken today?

That gap became the project.

What we learned

The most important lesson had nothing to do with code — it was about trust. Early on, we decided that no result would be believed just because an AI agent reported it. Every claim had to be backed by a command we could run ourselves, on evidence we could read ourselves.

That discipline paid off immediately. During real-world testing, our detector flagged what looked like a critical finding: a DROP POLICY statement in a real repository. Read as an isolated diff, it looked like a security control being ripped out. Read as a full commit, it was an idempotent migration — the policy was dropped and immediately re-created, unchanged. If we had trusted the tool's first answer, we'd have shipped a false positive as a headline result.

We learned that precision matters more than recall. A detector that never says anything wrong is more valuable than one that finds everything but cries wolf. Every false positive we caught this way — a database rename that looked like a removed predicate, a security check refactored into a helper function, a TLS-sounding flag name that meant something unrelated in a specific WebSocket library — became a permanent regression test. Our fixture didn't just grow; it grew scars, each one a lesson we'd never have to relearn.

We also learned that the network is never the problem, until it is. Late in the project, live scans against a verified real-world target started timing out — not because of rate limits, not because of Ratchet's code, but because a home network's HTTPS inspection was silently corrupting large file transfers. We diagnosed it the only honest way: by cloning the same repository outside our own tool, from a plain shell, and watching it fail identically. Sometimes the fix isn't a code change — it's admitting the bug isn't yours.

How we built it

Ratchet walks a repository's entire commit history and asks one question, over and over: did a security control exist, and then disappear?

The engine is built in layers, each one verified before the next was allowed to depend on it:

  • A frozen ground truth. Before writing a single detection rule, we built a small fixture repository with planted regressions and deliberately deceptive decoys — a column rename that preserves its predicate, a security check moved into a helper function, an idempotent migration disguised as a removal. That fixture was signed off by a human, then cryptographically frozen. From that point on, if a detector failed, we fixed the detector — never the test.
  • Six deterministic detection rules (R1–R8, minus two we deliberately didn't build) covering database policy weakening, stripped auth guards, widened trust boundaries, secrets crossing to the client, removed rate limits, and weakened input validation. No LLM judgment calls in the detection loop — every result is reproducible.
  • A correlation engine that links each weakening to its cause: either an earlier commit whose error message it was clearly responding to, or — the strongest signal — the weakening commit's own message, when a developer confessed the retreat themselves ("reverting to working state"). We proved this engine actually reasons about time, not just text, by shuffling commit timestamps and confirming the correlation score collapsed.
  • A still-present check that re-runs every detector against the live HEAD of a target, so a historical blip is labeled restored rather than presented as an open door.
  • DataHub integration that turns every finding into first-class metadata: a Dataset for the scanned repository, an Assertion for each rule, a DataJob for every cause and weakening commit, and — most importantly — a real lineage edge connecting cause to effect. Ratchet doesn't just read DataHub's graph; it writes back to it, exactly the way a tool should behave in an ecosystem meant to be built on.
  • A read-only MCP Server exposing three tools — list_recent_weakenings, get_finding, explain_cause — so any MCP-capable agent can query Ratchet's findings directly, without touching the scan pipeline.

We tested the whole pipeline against more than 50 real, independent public repositories — not samples chosen to flatter the tool. Every hit was verified by hand against the actual diff before being counted, the same way a security researcher would.

Note: Ratchet ships its own read-only MCP Server (ratchet-mcp) exposing

findings to any MCP-capable agent — this is a Ratchet-built MCP integration point, not a connection to DataHub's own MCP Server.

Challenges we faced

Distinguishing a real regression from something that only looks like one. Our biggest technical challenge wasn't finding weakened controls — it was not finding them where they didn't exist. Idempotent migrations, safe refactors, and library-specific naming collisions (a TLS-sounding flag that meant something entirely different in a specific WebSocket library) all had to be taught to the detectors, one real false positive at a time.

Keeping an AI-assisted build honest with itself. We built almost all of Ratchet with an AI coding agent — the same category of tool the project exists to audit. That meant applying the project's own philosophy to its own construction: an integrity guard that cryptographically froze the ground truth after human sign-off, so if the agent ever proposed loosening a test to make it pass, that action would be visible immediately, not silent.

Environment problems that looked like code problems. Windows-specific subprocess handling (orphaned git processes surviving a timeout), stale scratch clones piling up disk usage, and — right before filming the demo — a home network silently corrupting large file transfers over HTTPS. None of these were fixed by writing cleverer detection logic; they were fixed by refusing to assume the bug was where it was comfortable to look.

Knowing when to stop. During testing, Ratchet surfaced a live, self-confessed security regression on a real, active healthcare application — real patient data, fully exposed. It was the most compelling evidence we found. We're not showing it. Some findings are a responsibility, not a demo.

What's next

  • Two more detection rules (R2, R4) — ownership-predicate removal and inverted permission checks. Deliberately deferred; they require semantic judgment rather than deterministic pattern matching, and we chose to invest that time in broader real-world hunting instead.

  • Merge-commit-aware extraction. Ratchet currently diffs against a commit's first parent only, which can misattribute findings on merge-heavy repositories. A known, documented limitation scheduled for a proper fix.

  • Continuous monitoring, not one-off scans. A GitHub App / webhook integration that re-checks a repository on every push, so a regression is caught the moment it lands — not whenever someone remembers to run a scan.

  • Smart contract support. The same trajectory-based philosophy applies directly to Solidity: onlyOwner and nonReentrant modifiers removed under gas-optimization pressure, require() checks stripped to fit under contract-size limits. The failure mode is different — gas cost instead of AI silencing an error — but the detection shape is identical. Given that smart contract audits protect nine-figure sums, this is the highest-leverage extension of the engine.

  • Audit-drift detection for deployed contracts. Comparing a contract's last-audited commit against its live, deployed bytecode to catch unaudited changes that shipped after sign-off — a documented gap in the Web3 security industry that no existing tool automates.

  • Deeper DataHub-native workflows. Packaging Ratchet's detection logic as an official DataHub Skill (not just an SDK integration), so any DataHub deployment can adopt security-regression detection as a first-class capability rather than a bolt-on.

  • Exploitability-aware ranking. Beyond severity and still-present status, ranking findings by whether the affected file sits behind a public API route versus an internal admin path — turning a lead list into a prioritized attack surface.

Built With

Share this project:

Updates