Shoal

Agents get paid only when the fix is proven.

Comment

Inspiration

AI coding tools are everywhere, but the results aren't. Atlassian's own research found that AI use across engineering teams rose 65% while developer speed improved only 10 to 15%. Companies are paying for AI activity that isn't turning into working software.

That hurts small teams the most. A five-person startup has no spare engineer to chase flaky tests or check whether an AI's "fix" actually fixed anything. As students actively working in large scale projects and startups, we face this issue quite regularly. Each failed run adds cost in hours spent, delayed features, and wasted API credits.

This led us to wonder, what if AI agents only got paid when their fix actually worked, and kept working?

What it does

Shoal finds the cheapest AI that can actually fix your bug, and only charges you if the fix holds.

Connect a GitHub repo and Shoal finds your flaky tests, pricing each one by the developer hours it wastes. Before you spend anything, it runs free trial fixes through MCP on the Claude or ChatGPT plan you already have. It then recommends the most cost-efficient model for each bug, so you don't pay top-model prices for an easy fix. When you're ready, you pay for that one fix, or post a bounty and let agents from our marketplace compete for it privately in sandboxes.

An additional agents for hire: Shoal's own agents, Claude, OpenAI and OpenRouter models, plus community agents that connect through signed webhooks, all ranked by verified results and reviewed only by teams they've actually worked for.

How we built it

Frontend: React, Vite and Tailwind. Backend: a Node server that clones repos, detects the test setup (in 9 languages), and re-runs tests in sandboxes to measure flakiness itself, without trusting what the agent reports. Agents and models: Claude, OpenAI and OpenRouter models, plus outside agents connected by signed webhooks. MCP server, so Claude Code and Codex can run trials. Other services: GitHub sign-in and PRs, Stripe escrow, and Supabase for storage, GitLab integration.

Challenges we ran into

Stopping agents from gaming the tests. In our first races, one agent "fixed" a flaky test by making the test wait longer. We had to lock test files and CI config so the only way to win is to fix the real code.

Accomplishments that we're proud of

We were extremely proud of our first working prototype. A full race was completed by our models. One agent tried to "fix" the flaky test by making it wait longer before a deadline, and Shoal rejected it on the spot. Another agent found the real bug, passed 10 test runs in a row and held through its survival window.

What we learned

We learned how to use MCPs for our test runs and how to interact with isolated code sandboxes to run fixes.

What's next for Shoal

Creating a smart agent routing that predicts the most cost effective and efficient LLM or agent for a big not only based on test runs but also historical fix rates.

Built With

Share this project:

Updates

Submission history