Patchlane

Incident response for payment systems: Factory Droid patches the bug in a Daytona sandbox, a clean sandbox checks it with hidden tests, and an operator approves.

Comment

Inspiration

I run a small hosting company called Uniplex Host, and for about two years I've been the one who has to figure it out when a client's service goes down. A lot of that is incident response: working out what actually broke, then making sure the fix doesn't break something else. That's pretty much the work I want to do after school, in DevOps, platform or SRE roles, so for HackHarvard I wanted to build something for the incidents where the stakes are highest.

AI coding agents can find and fix a real bug in about a minute now. Banks still can't use them when they'd help most, which is during an incident that's moving customers' money. Nobody's going to hand an agent a shell on production, and reproducing an exploit means running attack code against the service.

So I wanted to see how far an agent could get if everything it touched was disposable. It gets a full copy of the broken service on an isolated machine with real tools inside it, and a separate check decides whether its fix is safe before a person approves it.

What it does

Patchlane is an incident responder for a bank's transfer service. The main incident is a missing ownership check: a customer sent 14 transfers out of three other customers' accounts, $7,546 in total, and every single one came back 201 Created.

  1. Detect. A fraud rule reads the transfer log and raises the alert.
  2. Reproduce. Patchlane creates a Daytona sandbox with no network, uploads the service, commits a git baseline, runs the failing test and replays all 214 transfers. Before any change, 14 of 14 attacks succeed.
  3. Fix. Factory Droid investigates using seven tools that only work inside that sandbox, writes a patch, and returns a structured report with the root cause, changed files, tests run and its confidence.
  4. Verify. Patchlane takes only Droid's source changes and applies them to a fresh copy in a second sandbox, along with held-out tests Droid never saw. After the patch, 0 of 14 attacks succeed and none of the 200 legitimate transfers get blocked.
  5. Approve. An operator reads the evidence and the diff and approves or rejects it. Both sandboxes get deleted either way.

Judges can also build their own incidents. The dashboard lists the 12 rules the bank service enforces, and you can tick up to three, pick how each one breaks from 34 options, and trigger it. I checked every combination ahead of time: 4,642 can run, and the 753 that can't show you why, like two bugs changing the same lines.

How I built it

  • Backend: a FastAPI orchestrator with a state machine and a live event stream to the dashboard. It runs in Docker as a Pterodactyl server behind nginx.
  • Frontend: React, Vite and Tailwind. Visitors can replay any of 12 real recorded runs. Live runs need an operator token, which gets checked with the server at sign-in.
  • Daytona: all of the agent's work happens in Daytona sandboxes built from a snapshot, created with network_block_all, labelled, auto-stopped and deleted at the end. The benchmark runs three incidents at a time, which came out to 48 sandboxes in under 8 minutes.
  • Factory Droid SDK: Droid runs with an allowlist of seven in-process MCP tools backed by the sandbox. Its 19 host tools are turned off and its working directory is an empty temp folder outside the repo. Before sending the prompt, Patchlane checks list_tools() and list_skills() and refuses to run if anything on the host side is enabled. A permission handler cancels and counts any other request, and the final answer comes back as a typed RemediationResult.
  • Chalk: every benchmark run uploads its rows as a Chalk dataset and runs a new evaluation with five scorers: a custom function, two Chalk SQL scorers, a built-in contains, and an llm_judge that goes through Chalk's model router. Patchlane combines Chalk's per-row scores like this:

$$\text{composite} = 0.40\,f + 0.25\,r + 0.15\,t + 0.10\,c + 0.10\,j$$

where f is the share of held-out tests that pass, r is no regressions, t is no tampering, c is correct file and j is the judge's root-cause score. The weights can't override the gate:

$$\text{safe to review} \iff f = r = t = 1$$

Results. I ran each of the 12 seeded bugs twice through the full pipeline. All 24 runs verified, Chalk scored every row 1.0 on all five scorers, 176 of 176 held-out tests passed, and the median run took 54 seconds. The 22 alternative bugs verified in 43 of 44 runs, and the one that didn't stopped because a Daytona command hung. Sampled scenarios verified 12 of 12 with two bugs and 8 of 8 with three, and a three-bug scenario I triggered from the dashboard verified in 64 seconds with 29 of 29 held-out tests passing. The backend has 588 tests, and 519 of them cover the scenario builder.

Challenges

  • Locking Droid down. My first isolation check passed even though host tools were still on, because MCP tools are named daytona___<tool> and my filter missed them. Then I found out Droid 0.244 loads MCP tools through ToolSearch and calls them from inside a Script runtime, where permission prompts don't work. So I made the allowlist the actual security boundary, checked that Script has no modules, process access or fetch, and made the isolation check refuse to run if anything doesn't match.
  • Streams closing themselves. The SSE library ended every stream whenever Droid's internal server shut down. I swapped it for a plain streaming response with 15-second pings.
  • Bugs hiding each other. Once judges could combine bugs, some pairs masked each other, so with both in place one of the incident tests would pass. I wrote a validator that merges every combination with git merge-file and runs the tests. It went through 525 pairs and 4,870 triples in about 16 minutes, and it keeps a plain-English reason for every combination it refuses.
  • Getting Chalk running. Evaluations sat in pending until Chalk staff turned them on, and after that a capacity limit had to be raised. The LLM judge only worked through Chalk's router with one specific model and max_tokens set.
  • Hosting. Pterodactyl runs the image as its own user and passes the port in an environment variable, so I had to rewrite the entrypoint. nginx on the same machine resolved the node's own hostname to 127.0.1.1, which gave me a 502 until I pointed it at the IP.

What I learned

  • Droid's config said it was locked down when it wasn't. Checking the live tool list at runtime is what actually caught it.
  • Git ended up being the most reliable way to tell what the agent changed. The diff against the baseline drives the dashboard, the tampering check and the patch that goes to verification, and none of that depends on what Droid says it did.
  • Held-out tests need over-correction cases too, like "the owner can still transfer." Without those, a patch that just blocks every transfer would pass.
  • Writing the benchmark early meant I could measure every change after it. Every new bug and variant went through real runs before I offered it to anyone.

What's next

Next I'd like to point Patchlane at a real service's repository and test suite. I also want it to handle several incidents at once, with live runs scored in Chalk alongside the benchmark.

Built With

Share this project:

Updates

Submission history