Inspiration
"The hard part of AI coding isn't getting a fix. It's knowing whether to believe it."
|
|
| 😩 The problem |
Every team has a pile of small bugs and "quick tweak" tickets nobody has time for. |
| 🤖 The catch |
AI can write a fix in seconds, but a confident-looking patch that quietly breaks something else is worse than no patch. |
| 💡 Our idea |
Don't ask people to trust the AI. Make real checks the judge, and take away the AI's power to merge. |
We kept seeing the same three things go wrong with AI-generated code:
| What we saw |
What we decided |
| ❌ Fixes that look right but fail in edge cases |
✅ A fix only counts if it passes real checks in a sandbox |
| ❌ Tools that can push or merge on their own |
✅ Agents cannot merge. A human always does |
| ❌ Logs that are a wall of text nobody reads |
✅ A live, playful view where you can see what each agent is doing |
That's how PatchLoop was born: AI agents propose. Tests decide. Humans merge. And it's why it has a mascot, Patchy 🤖🩹, and a live Agent Town instead of a terminal dump.
What it does
You type a task in plain English, pick a repo, and a swarm of AI agents does the work, then has to prove it.
YOU YOU
│ │
▼ ▼
┌────────┐ ┌────────┐ ┌─────────────┐ ┌─────────┐ ┌────────┐
│ 1 Task │──▶│ 2 Plan │──▶│ 3 Swarm of │──▶│ 4 Gates │──▶│ 5 PR │──▶ 👩💻 Review & Merge
│ in │ │ units │ │ agents work │ │ in a │ │ with │
│ English│ │ + deps │ │ in parallel │ │ sandbox │ │ proof │
└────────┘ └────────┘ └─────────────┘ └─────────┘ └────────┘
│
❌ fails a gate? the candidate is thrown out
| # |
Step |
What happens |
| 1️⃣ |
Describe |
Plain-English task: a bug, a feature, a refactor, a UI change |
| 2️⃣ |
Plan |
The planner reads the repo and splits the work into units that can depend on each other |
| 3️⃣ |
Swarm |
Agents work in parallel. Helpers locate the right files and write a test that reproduces the bug |
| 4️⃣ |
Verify |
Every candidate runs in a sandbox against hard gates (build, tests, policy, security) |
| 5️⃣ |
Ship |
Passing units are merged, re-checked, and opened as a pull request with the evidence |
🛡️ The guarantees
| Promise |
How it's enforced |
| 🚫 Agents never merge |
There is no merge ability in the agents. A human merges, with their own GitHub permissions |
| ✅ Tests are the judge |
A fix counts only when the sandbox checks pass |
| 🔒 Safety on every candidate |
A policy gate and a security scan are part of the hard gates |
| 🧾 Honest results |
A run that skipped a check is never shown as a pass |
| 💸 Bounded cost |
Every run has a token and time budget cap |
🧾 Five honest outcomes
| Outcome |
Meaning |
| 🎉 Complete |
Every hard gate passed |
| ⛅ Partial |
Work landed, but some checks couldn't run. The notes say which |
| 🌧️ Failed |
The checks say exactly what went wrong |
| 🛑 Blocked |
A policy or safety rule stopped it before any work landed |
| 🤔 Needs input |
The task was too vague, so the team asks questions first |
🧰 Three ways to use it
| Interface |
Best for |
Example |
| 🌐 Web app |
Watching the swarm live in Agent Town |
Start a task, watch agents take their desks |
⌨️ CLI (patchloop) |
Developers who live in the terminal |
patchloop fix --repo octo/widgets "pagination is off by one" |
| 💬 Slack |
Getting a ping when a run finishes |
Completion messages in your channel |
Plus FBA 🤖, a helper chatbot that answers questions about PatchLoop right on the site.
How we built it
🏗️ Architecture
🌐 Browser ─┐
⌨️ CLI ─┼──▶ Backend (Go :8080) ──── secret header ────▶ Agent service (Python :8001)
💬 Slack ─┘ │ │
│ ├─▶ 🧠 Models (Nebius)
▼ ├─▶ 📦 Sandboxes (Contree)
🗄️ MongoDB └─▶ 🔍 Traces & logs
│
└─▶ 🐙 GitHub App (read repos, open PRs)
🧰 Tech stack
| Layer |
Technology |
Why |
| Frontend |
Next.js 16, React 19, Tailwind v4, framer-motion |
Fast pages, themeable tokens, springy animations |
| Auth |
Clerk |
Sign-in and profile without building our own |
| Backend |
Go + MongoDB |
Simple, fast API; stores runs and settings |
| Agents |
Python + LangGraph |
Orchestrates planner, workers and reviewers |
| Models |
Nebius-hosted LLMs |
The agents' brains |
| Sandbox |
Contree |
Runs untrusted candidate code safely |
| Repo access |
GitHub App |
Scoped access, opens PRs |
| CLI |
Go + Cobra |
One fast binary with a live swarm view |
| Observability |
New Relic, LangSmith, JSON logs with trace IDs |
Follow one run across every service |
| Evaluation |
DeepEval, Ragas, custom metrics |
Measure the system, not just hope |
🐝 Inside the swarm
| Piece |
Job |
| 🧭 Planner |
Splits the task into units with dependencies |
| 🔎 Locator |
Finds the files that matter |
| 🧪 Reproducer |
Writes a test that fails on the original code, so we know the bug is real |
| 🛠️ Workers |
Build candidate fixes in parallel |
| 🧺 Merger |
Combines passing units with a three-way merge |
| 🚦 Gates |
Build, tests, policy, security |
| 🔋 Budget |
Caps tokens and time |
🧪 How we measure it
| Fact |
Value |
| Golden test cases |
30 |
| Integrity checks (reference fixes really pass, originals really fail) |
175, all passing |
| Metric families |
Deterministic (offline), DeepEval, Ragas |
| Safety gate |
Regression gate that fails the run if scores drop |
What the 30 cases cover:
Bug fix, single file ████████████████████████████ 9
Safety / guardrails ██████████████████████ 7
Bug fix, multi-file █████████ 3
Locating the right files █████████ 3
Writing the reproducer ██████ 2
Unsolvable / vague tasks ██████ 2
Follow-up changes ██████ 2
Pull request quality ██████ 2
─────────────────────────────────
30
Honesty note: the offline "fake mode" uses scripted agents. It proves the harness works (dataset, metrics, gate, reports). It does not claim to measure model quality, and every report says so.
🎨 Design
| Feature |
Why |
| 🤖 Patchy, the round blushing bot with a bandage |
Gives a serious tool a friendly face |
| 🏘️ Agent Town |
Desks, speech bubbles and gate lights show real events live |
| 🌈 Pastel theme |
Sticker borders, pill buttons, bouncy springs |
| ♿ Accessible |
Text colours checked against WCAG AA; motion turns off for people who ask for reduced motion |
Challenges we ran into
| # |
😵 Problem |
🔍 Symptom |
🔧 Fix |
| 1 |
Repo already broken |
Agents burned the whole token budget on a build that failed before any change, and the run was wrongly labelled "blocked" |
Added a repair step that fixes the baseline first, dropped lint checks that can't work in a sandbox, and report empty runs as "failed" |
| 2 |
Wrong sandbox image |
npm: not found on a Node repo with no test script |
Choose the sandbox image from the repo's files |
| 3 |
Demo repos unreachable |
404 on bundled swarm demos; protocol error from an empty GitHub token |
Wired demos into the loader; send the auth header only when a token exists |
| 4 |
A wall of agent logs |
The run page was unreadable |
Rebuilt it as Agent Town, mapping real events to a live scene with nothing faked |
| 5 |
One bad line in .env |
The backend silently loaded no variables |
Found the stray line, fixed it, and learned to make config failures loud |
| 6 |
Silent monitoring failure |
Logs never reached New Relic |
Traced it to the wrong key type, and documented the right one |
| 7 |
Flaky network |
Live runs died with confusing sandbox errors |
Found a failing DNS resolver; plan to show a clearer "can't reach Nebius" message |
| 8 |
Click target too small |
Run history rows only opened from the title text |
Made the whole row clickable |
Accomplishments that we're proud of
|
Accomplishment |
Why it matters |
| 🛡️ |
Agents that can't merge, by design |
Safety isn't a setting someone can switch off; a human is always in the loop |
| ✅ |
Tests as the judge |
Every change arrives with proof: build, tests, policy and security results |
| 🐝 |
A real parallel swarm |
Planner, locator, reproducer, workers and merger working together within a budget |
| 🧾 |
Five honest outcomes |
The product never dresses up a partial or failed run as a success |
| 🏘️ |
Agent Town |
You can actually watch agents spawn and work, driven by real events |
| 🧪 |
A real eval suite |
30 golden cases and 175 integrity checks, all passing, with a regression gate |
| 🧰 |
Three interfaces |
Web, CLI and Slack all drive the same engine |
| 🎨 |
A full cute re-theme |
Playful pastel design that still passes WCAG AA contrast |
| 🎬 |
Delight in the terminal |
A big animated Patchy greets you when patchloop login succeeds |
What we learned
Each lesson follows the same pattern: what we assumed → what actually happened → what we do now.
| # |
Theme |
Key lesson |
| 1 |
🤖 AI agents |
The verifier matters more than the model |
| 2 |
🧾 Honesty |
A truthful label beats a pretty one |
| 3 |
🧪 Evaluation |
Test the measuring stick before measuring |
| 4 |
🎨 Design |
A theme is a system, not a paint job |
| 5 |
🔧 Operations |
Silent failures cost the most time |
1. 🤖 AI agents: the verifier matters more than the model
|
|
| We assumed |
A smarter model would mean better fixes |
| What happened |
Agents wrote convincing patches that still failed real tests, and spent their whole budget on a build that was broken before they touched it |
| What we do now |
Every candidate must pass hard gates in a sandbox. A broken baseline is repaired first, so agents aren't blamed for old bugs |
Takeaway: a fix is only as trustworthy as the checks that judge it.
2. 🧾 Honesty: a truthful label beats a pretty one
|
|
| We assumed |
Pass or fail was enough |
| What happened |
Runs where checks were skipped looked like successes, and runs that changed nothing were labelled "blocked" when they had really failed |
| What we do now |
Five distinct outcomes, each with its reasons shown on screen |
Takeaway: users trust "partial" more than a fake "complete".
3. 🧪 Evaluation: test the measuring stick before measuring
|
|
| We assumed |
We could jump straight to scoring the model |
| What happened |
Without checking the harness first, a bad dataset or a broken metric would have produced numbers that looked real but meant nothing |
| What we do now |
A three-step ladder, each step proving the one below it |
| Step |
What it proves |
Needs keys? |
| 1️⃣ Integrity check (30 cases, 175 checks) |
Reference fixes really pass; original code really fails |
No |
| 2️⃣ Fake mode (scripted agents) |
The pipeline, metrics, gate and reports work |
No |
| 3️⃣ Live mode (real models) |
How good the agents actually are |
Yes |
How we score "solved within k tries", using the standard unbiased estimator:
$$\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$
| Symbol |
Meaning |
| $n$ |
attempts made |
| $c$ |
attempts that passed |
| $k$ |
attempts we allow |
Takeaway: fake mode proves the harness, never the model, and every report says so.
4. 🎨 Design: a theme is a system, not a paint job
|
|
| We assumed |
Re-theming from dark to pastel meant swapping a few colours |
| What happened |
Over a hundred hard-coded colours (like white text) were hiding outside the theme and would have vanished on a light background |
| What we do now |
All colours are tokens, and every text and background pair is checked for contrast |
Takeaway: "cute" only works when it is also readable and honest.
5. 🔧 Operations: silent failures cost the most time
| What broke |
Why it was hard to find |
What we learned |
One stray line in .env |
The backend said "no .env found" instead of naming the bad line |
Error messages must name the real cause |
| Monitoring sent nothing |
The wrong key type was accepted without any warning |
"Enabled" is not the same as "working" |
| Live runs failed randomly |
A flaky DNS resolver surfaced as sandbox errors |
Network errors need plain-language messages |
| Run history felt broken |
Only the title text was clickable |
Small UX gaps feel like bugs to users |
Takeaway: make every failure loud, specific and early.
What's next for PatchLoop
| Horizon |
Goal |
| 🟢 Now |
Publish real eval numbers from live model runs |
| 🟡 Soon |
Browser-capable sandboxes so UI and visual checks can run, turning more "partial" runs into "complete" |
| 🔵 Later |
Clearer network-error messages, replayable run timelines, more task types |
🩹 PatchLoop: let the agents try, let the tests judge, let people decide.
Log in or sign up for Devpost to join the conversation.