Field Tested

The setups users post in issues become release tests. GitLab Duo harvests them, and GitLab CI replays all 29 on each linkding candidate before I promote. The replay caught the real 1.43.0 break.

Comment

Inspiration

In September 2025, linkding, a self-hosted bookmark manager, released 1.43.0. A clean upgrade worked for the maintainer, who wrote "I can't reproduce this." Some users' container configurations still pointed PATH at the virtualenv the release had moved, and their containers went into a restart loop. The issue thread ran to 38 comments over about six weeks.

One of those configurations had been in linkding's own tracker for fourteen months. Issue 787, a bug report about memory use, contains the Compose file that breaks on 1.43.0. Users post exactly how they deploy when they ask for help, and I wanted those posts to count as tests before the next release.

A GitLab Duo flow turned the user's post (issue #27) into a typed profile in merge request !22

What it does

Field Tested turns the deployment setups users post in issues into release tests. Before, linkding's maintainer learned about this break from the comments after the release. With Field Tested on GitLab, every admitted setup is replayed against the release candidate before promotion.

  1. Harvest. I mention a custom GitLab Duo flow on an issue. It reads the issue and its comments, writes a typed profile for each runnable docker run command or Compose service it finds for the project's image, links each profile to the post it came from, and opens a merge request.
  2. Review. Before I merge a profile, GitLab CI checks it. A cross-check compares the profile's fields with the text it copied, and that text with the user's real post, line by line. A scrubber redacts secrets, and an admission run checks the setup starts on the current release. The source check confirms the copied lines and their order; it cannot show that every setting in the post was copied, which is one reason I review each profile.
  3. Replay. On every release candidate, GitLab CI builds the image and replays every admitted setup on the last release and on the candidate. Each runs on its own Docker network with no route out, and an egress canary with a positive control fails the run if that isolation cannot be shown.
  4. Gate. A release statement says how many setups from how many issues pass, names each one that fails with a link to the post it came from, and lists the setups it could not replay with the reason. A failure turns the pipeline red and keeps the manual promote job from running.
  5. Respond. A second Duo flow, mentioned on the blocked merge request, reads the failed replay and the diff, explains the failure, writes a release-note line, and says what it observed and what it inferred. When it sees a small compatibility change, it opens a merge request with the fix, and CI replays it like any other candidate.
  6. Promote and deploy. When everything passes, I run the promote job. GitLab CI copies the tested digest to Artifact Registry and deploys it to the existing Cloud Run service as a new revision with no traffic. Traffic moves only after that revision passes the same checks on its own URL. If the checks after the move fail, the job puts traffic back on the previous revisions. (The very first deploy creates the service, so it serves before its checks run.)

CI checks the changes Duo proposes. I decide whether to proceed at three points: merging each profile, merging a fix, and promoting the release.

The real linkding 1.43.0 change: 28 of 29 setups pass, the one from issue 787 fails, and promotion is blocked

How I built it

GitLab Duo Agent Platform. Two custom flows in the AI Catalog, defined in .gitlab/duo/ and triggered by mentions. The harvester uses get_issue, list_issue_notes, create_file_with_contents, run_command (git only) and create_merge_request. The responder reads the failed child pipeline with get_failing_bridge_jobs, get_downstream_pipelines and get_job_logs, reads the change with list_merge_request_diffs, and answers with create_merge_request_note. Both prompts tell the flow to treat issue and merge request text as untrusted data.

GitLab CI. A generated child pipeline runs the egress canary, shards the replay across six Docker-in-Docker jobs, and ends with the statement and gate. Image digests pass between jobs as dotenv artifacts, and results appear in merge requests as JUnit reports. Promotion is a manual job on main, and the Cloud Run deploy is recorded as the production environment.

The deterministic core. field_tested/ is plain Python (PyYAML is optional, for one YAML check). Its compiler builds a docker run argument list from an allow-list of fields, never a shell string. It rejects privileged options, never mounts a host path or socket (a host directory becomes a fresh empty volume), replaces redacted secrets with a placeholder, and records what it left out, such as env files whose contents were never posted. 281 unit tests.

Google Cloud. Cloud Run runs the deployed linkding and Artifact Registry holds the promoted image. GitLab CI signs in through Workload Identity Federation with no service-account keys, and the provider accepts only this project's protected main branch and the production environment.

Cloud Run: the new revision passes its checks with no traffic, then takes 100% of traffic

Results

All runs are on GitLab's shared runners, with images pinned by digest.

  • From 35 linkding issues, Duo proposed 38 profiles in 29 merge requests. I merged 25 unchanged and 3 with a recorded interpretation, and corrected 1 before merging it. Of the 38, 29 setups from 22 issues were admitted; 1 also fails on 1.42.0, 4 were rejected by the compiler and 4 were not replayed, each with the reason.
  • The replay of the upstream 1.43.0 packaging change (!31) passed 28 of 29. The setup from issue 787 failed with exit code 127, and promotion was blocked. The responder's fix, keeping the old path as a link (!33), passed 29 of 29; after I merged it into the candidate branch, !31 passed too.
  • A planted regression (!32, labeled as planted) serves static files from a missing directory. Every setup stayed healthy and all 29 failed the "stylesheet is served" check, so a health-only smoke test would have passed it.
  • 37 of 38 profiles pass a check that every copied line occurs in the user's post, in order; the one flagged has a comment line the model added. A planted edit that changes a profile and its copied text together (!34) fails the same check.
  • On bentopdf, 16 issues gave 13 profiles, none needing a correction. The replay of release 1.16.1 passed 6 of 7 admitted setups. Issue 427's hardened setup, posted the evening before the release, failed with exit code 1. I found no report of the break in the tracker for 53 days, and on 21 March 2026 its author wrote: "Alright looks like I didn't notice but mine biffed it too."

Each admitted setup replays on the last release and on the candidate, each on a network with no route out

Challenges I ran into

  • Models change what they copy. In research they resolved variables, fixed users' typos and left default passwords in the profiles. So each profile keeps the user's text, is checked against the original post and scrubbed, and still needs a maintainer to merge it.
  • Each replay runs a stranger's configuration, so it runs on an internal Docker network checked by the egress canary, with memory, CPU and process limits. Writable storage is not capped, so a hostile setup could fill its own job's disk; that risk is documented.
  • Coverage is limited to amd64 containers and the configured health and HTTP checks. Replays use fresh volumes and placeholder secrets, so host behavior and files users never posted are not reproduced, and each setup's notes say what was left out. Profiles the compiler rejects and image variants that cannot be built are listed separately, with the reason, and are not counted as covered.
  • My role cannot enforce "pipelines must succeed" (HTTP 403), so a failing field test blocks the promote job but cannot by itself stop someone with merge rights. Child pipelines started from merge request pipelines also needed workflow: rules: when: always before GitLab would create them.

Accomplishments that I'm proud of

  • A suite that Duo harvested from issues reproduced the real linkding regression: exactly one of 29 setups failed on the upstream change, the one from issue 787.
  • A historical replay on a second project found bentopdf's 1.16.1 break from a setup posted the evening before the release.
  • The responder's fix is one symlink, and CI replayed it against all 29 setups before I merged it.

The responder flow's fix in !33: keep the old virtualenv path as a link

What I learned

  • The source check flagged a comment line the model had added to one profile and caught the planted edit in !34.
  • Health checks alone would have passed the planted static-file regression; the HTTP check caught it in all 29 setups.
  • The responder was more useful when it separated what it observed in the logs from what it inferred about the cause.

What's next for Field Tested

  • Fixture files and host variants (ARM runners, rootless Docker) so more posts become replayable.
  • A pipeline-event trigger so the responder runs as soon as a field test fails, without a mention.
  • Packaging field_tested as a CI/CD component that any project can include.

Credits

Music in the video: "Wallpaper" by Kevin MacLeod (incompetech.com), licensed under Creative Commons: By Attribution 4.0, https://creativecommons.org/licenses/by/4.0/

Built With

  • artifact-registry
  • docker
  • gitlab
  • gitlab-ai-catalog
  • gitlab-ci
  • gitlab-container-registry
  • gitlab-duo-agent-platform
  • gitlab-environments
  • google-cloud-run
  • python
  • workload-identity-federation
Share this project:

Updates

Submission history