Hold Water for READMEs

README promises become failing tests, agents fix them, and GitLab CI alone decides what merges.

Comment

Why I made this

Every README makes promises. "Orders over $100 get 10% off." "Only admins can edit products." "Bad input never gives a 500 error." Nobody checks these once the code starts changing. And when an AI agent is the one changing the code, it gets worse, because now nobody checks the agent either.

So I asked a simple question. What if the promises in the README were the tests? And what if the thing that decides what gets merged was not an AI at all, but the CI pipeline?

That is Hold Water for READMEs. The idea is in the name: does the README hold water? The whole path is README claims, then failing tests, then gated merges, and all of it runs inside GitLab.

What it does

You write a normal issue in plain English. After that, the loop runs without you.

Issue created
   |
   v
Builder agent opens a merge request (builder/ branch)
   |
   v
CI gates: guard_protected, unit_tests, repro_tests, security_gate
   |-- all green, no reproduction tests yet --> Breaker commits failing tests
   |-- a gate is red before the Breaker ran --> label "Needs human review"
   v
CI gates run again
   |-- a gate is red --> Fixer patches the branch (2 calls at most) --> gates again
   |-- guard fails, or two failed agent calls --> label "Needs human review"
   v
All green: GitLab merges the merge request
   |
   v
main: package_image --> smoke_test (staging soak, 100 s) --> promote_image (tag "stable")
   |-- unhealthy after the soak: not promoted, postmortem issue opened

The three agents are custom flows on the GitLab Duo Agent Platform. They never call each other. A small CI job called call_agents decides who runs next and posts one mention on the merge request. The decision is a plain function, so it can be unit tested, and it is.

Part What it is Its job
Builder Duo flow Turns an issue into a merge request
Breaker Duo flow Reads the README claims, starts the real app, commits one failing test per broken claim
Fixer Duo flow Fixes only what has a failing test, on the branch, never on main
call_agents CI script Decides who runs next, max 2 calls per agent
guard_protected CI script Stops agents from touching pipeline config, scripts, the dashboard, or existing tests
unit_tests, repro_tests, security_gate CI jobs Decide whether the code may merge
auto_merge CI script Turns on merge when green
score.py CI script Computes the Ship-Readiness Score from CI results

After the merge, the build goes into the GitLab container registry. It runs as a staging container inside CI, gets checked, sits for 100 seconds, and gets checked again. Only a healthy build is tagged stable.

The demo app and its promises

The app is called ShopNest. It is a small store API (FastAPI and SQLite) that I wrote for this. Its README makes nine promises, and the Breaker turns each one into tests.

ID Promise
C1 Orders with a subtotal over $100 get a 10% discount. Smaller orders get none.
C2 Stock never goes below zero. An order that cannot be filled gets a 409.
C3 Only admins can create, edit or delete products.
C4 A user can only see their own orders.
C5 Invalid input returns a 4xx, never a 500.
C6 At most one coupon per order. More than one gets a 400.
C7 Money values are rounded to 2 decimals.
C8 Search input is treated as data, never as SQL.
C9 No secrets are exposed over HTTP.

What happened when I ran it

Every number below comes from CI, the GitLab API, or a file in the repository. Where the history does not say, the dashboard shows [not recorded] instead of a guess.

What I measured Result
First run on the untouched app (18 Oct) 8 of 9 claims broken, 17 failing reproduction tests, score 45 out of 100
10 benchmark bugs, 4 of them held out from the agents Original app 0 of 10. After the loop 10 of 10, but I count it as 9 of 10 because the Builder fixed one of them by accident while building a feature
Builder merge requests merged 15 in total, 11 hands-off and 4 supervised
Ship-Readiness Score on main 100 out of 100, computed by a script, never by a model

The Ship-Readiness Score is out of 100: intent claims passing (40), security (25), tests (20), deploy health (15). If a part has no data, it is left out and the rest are re-weighted, so the score never shows something the build did not earn.

Seven fresh issues, one at a time

On 22 October I gave the system five new issues, and on 25 October two more. I logged every outcome, including the ones that needed a person. A run counts as hands-off only if nobody touched the merge request.

Run Issue Asked for Result Hands-off Minutes to merge
R1 #16 Optional sku field on products Merged (!18) yes 15.9
R2 #17 min_price and max_price filters Merged (!19) yes 13.3
R3 #18 GET /orders/{id}/receipt Merged (!20) yes 9.9
R4 #19 Admin stock adjustment endpoint Merged (!21) yes 15.5
R5 #20 Product tags with a tag filter Merged after a human step (!22) no not comparable
R6 #22 GET /products/count Merged by hand before the Breaker finished (!24) no not comparable
R7 #23 Optional rating field (title only, no description) Merged (!25) yes 10.8

R5 got stuck. The Builder added a tags field, and that broke an old test that checked the exact list of response fields. Existing tests are protected, so no agent was allowed to edit it, and the loop just waited. I had to edit the test myself and add the allow-protected label. That showed me two real problems, which I fixed (see the table of mistakes below).

Two bad deploys

I wrote a patch that makes /health start failing about 90 seconds after the app starts. Every unit and reproduction test still passes, so the merge request merges. Only the staging soak can catch it.

Bad build What happened on main Recovery
Test 1 MR !16 Pipeline #104: smoke_test failed after the soak, image not promoted, issue #14 opened. The pipeline looked green only because those jobs were still allowed to fail Revert, pipeline #106 healthy and promoted. Then I made the three staging jobs required
Test 2 MR !17 Pipeline #120 went red: smoke_test failed, promote_image skipped, issue #15 opened Revert, pipeline #122 green, staging healthy, promoted

Test 2 matters because it shows a failed staging check really does turn main red now. The revert in both cases was done by hand. I call it a verified revert, not an automatic rollback.

How it keeps agents honest

README text, issue text and code are all treated as untrusted. The rules live in the pipeline, not in prompts.

  • Agents cannot change .gitlab-ci.yml, .gitlab/, scripts/, dashboard/, Dockerfile, CODEOWNERS, pytest.ini or conftest.py at any depth.
  • Existing files in tests/reproductions/ and tests/unit/ are append-only. The guard checks commit by commit, so a later commit cannot quietly rewrite a test that an earlier commit added.
  • The label allow-protected lifts the guard, but only if a human with Maintainer access (or the project owner) added it. The pipeline asks the API who added the label. Bots are refused, and if the check fails, there is no bypass.
  • security_gate blocks the merge while a reproduced exploit is still unfixed.
  • Each agent gets at most two calls per merge request. After that, a person is asked.
  • I tested prompt injection three times, once per agent (a hidden instruction in the README, in an issue, and in a Breaker report). That is three samples, not a proof.

How I built it

I used the GitLab Duo Agent Platform for the three flows, with triggers on "work item created" (Builder) and on mention (Breaker and Fixer). The rest is GitLab CI/CD: the guard, the test gates, packaging with kaniko into the GitLab container registry, the staging check as a CI service, and promotion with crane. The dashboard is a single HTML file on GitLab Pages, and its numbers are rebuilt from CI data on every merge. The flows are frozen at the tag v1.0-freeze, so the prompts did not change after that. Tests, docs and scripts did.

What went wrong, and what I changed

What happened What I changed
Pipeline-event triggers were unreliable and agents did not always start A CI job now posts the mention itself, in a way that always behaves the same
A bad deploy showed a green pipeline (#104) The three staging jobs are required, and I proved it with a second bad deploy (#120)
R5 stalled silently when a gate went red before the Breaker ran call_agents now adds the label Needs human review
The override label did not work in R5, because I am only a Developer in my own project The guard also accepts the project owner by user id, and prints why a bypass was refused
The Breaker writes tests that assert an exact list of response fields, so any new field breaks them Written down as a finding. I did not change it because the prompts were frozen
One benchmark bug (C6) was fixed by the Builder, not by the loop The benchmark is quoted as 9 of 10
A cleanup commit removed a script the staging job needed, and main went red Restored the script, closed the postmortem issue (#21)

What I learned

The more decisions I moved out of the agents and into the pipeline, the more I could trust the results. The agents write and repair. The pipeline decides. I also learned that small honest numbers are worth more than big vague ones, which is why the failed run (R5) and the two bad builds are on the dashboard and not hidden.

Honest limits

  • I wrote the demo app and its bugs, and the 10 benchmark bugs too. This is not an independent benchmark.
  • It has only run on one stack, Python and FastAPI. The docs explain how to adapt it, but I have not tried another stack.
  • The samples are small: five fresh runs, two bad deploys, one benchmark run per checkout.
  • Agent mentions are posted by a CI script using the project owner's token, because GitLab does not let bot accounts start flows. A person's token is behind every mention.
  • Deployment means a staging container inside CI plus the GitLab container registry. Nothing serves real traffic. I drafted a Cloud Run path, never ran it, and removed it, so there is no Google Cloud deployment.
  • I do not have token cost per run, so I do not report it.
  • The score only measures what the tests measure. A promise that nobody wrote a test for is not in it.

What I would do next

Deploy to a real cloud target, try a second stack, ask someone else to write the benchmark bugs, make the soak longer than 100 seconds, and make the Breaker write more tolerant assertions so new fields do not break old tests.

The live dashboard has every number above, and each one links back to its source. docs/JUDGES.md in the repository is a five minute path to check the claims yourself.

Built With

Share this project:

Updates

Submission history