ColdStart

A GitLab Duo agent that runs your README's setup in a clean CI container, fixes what breaks, and hands you a verified README and a Dockerfile to merge.

Comment

Inspiration

A README is a claim that a project runs, and nobody checks it until a newcomer tries. Versions drift, build steps go missing, an env var gets assumed. After the code is written, this is the first thing that breaks for every new contributor, reviewer and deploy target.

What it does

Comment @cold-start verify on a merge request in a project where the flow is enabled.

  1. A supervisor agent decides what the mention means and delegates to a setup agent, which reads the README, manifests, version files and CI config, plans the setup, writes .coldstart/ on a new branch, and opens a draft merge request.
  2. GitLab CI is the clean container. The pipeline runs the plan step by step with machine-readable markers.
  3. If it fails, a diagnose agent reads the failing job's log, classifies the failure, quotes the real log line as evidence, commits the smallest setup-layer fix (a runtime image, a package, a build step), and explains it in a note. Maximum five attempts.
  4. When it passes, a finalize agent adds a "Getting started (verified by Cold Start)" README section, a Dockerfile, and a coldstart:package CI job that proves the Dockerfile builds. It posts a verdict, marks the MR ready, and lists README lines the run disproved. A human merges.

It never edits application source. If the failure is an app bug or needs a real secret, it stops with needs a human, names the missing variable, and invents nothing.

With an optional auto-retry job, one comment runs the whole loop (tested end to end in about 6 minutes on a private test project).

Autonomy level

Path A (start fresh), Supervised: the agents run the multi-step loop on their own within a budget of five fix attempts (setup, run, diagnose, fix, finalize, package), and a human approves the outcome by merging. Every change is a visible commit and note. In the default mode a person comments retry between stages; with the optional one-comment mode the pipeline posts those comments, so the person only starts the run and approves the result.

Stages of the lifecycle covered

Plan (setup plan), build and test (CI runs install, build and the probe), fix (diagnose loop), package (Dockerfile built in CI), document (verified README section), review (draft MR and a human merge). Deploy is not implemented yet.

How I built it

  • A GitLab Duo custom flow (Flow Registry v1) in supervisor mode with three sub-agents.
  • A static Console (no build step, GitLab Pages) that reads GitLab's public API and replays runs, with tests for the run parser.
  • GitLab CI for the clean container, with a step-marker contract (COLDSTART STEP n START|OK|FAIL) between run.sh and the diagnose agent.
  • A static checker (scripts/check_flow.py, run in the repo's own CI) that catches the flow mistakes the catalog rejected or the runtime mishandled.
  • Seeded failure fixtures (test-repos/fixtures/) so the tests are reproducible.

Results

All five runs below were re-run on a public project, cold-start-demo, so every merge request, pipeline and commit opens without a login. Each is also replayable in the Console.

  • Seeded Node 14 break (MR !2): failed on node:14 with ReferenceError: structuredClone is not defined, diagnosed from that real log line, fixed to node:18 in one attempt, passed, then added the README section, the Dockerfile and the package job, with the verdict "runs with fixes". It also flagged the README's "Requires Node 14" and .nvmrc as wrong, without editing them.
  • Needs-a-human case (MR !3): stopped, quoted Error: SERVICE_TOKEN is required, changed nothing, named the variable and asked for a masked CI variable.
  • Three unmodified public repos, passed first try, each with a Dockerfile proven to build by the coldstart:package job: expressjs/cors (MR !4, 49 tests), pallets/itsdangerous (MR !5, 297 tests) and kennethreitz/records (MR !6, 31 tests). All three READMEs lack contributor setup steps, so the agent inferred every step (from package.json, the lock file, or the repo's own CI workflow) and the verdict says so: "runs, but the README does not document these steps".
  • Each run in the public project was driven with two or three comments (verify, then retry); auto-retry was not enabled there. Auto-retry was tested end to end on my private test project, where one comment ran the whole loop in about 6 minutes.
  • Other seeded breaks (missing build step, missing system package, a removed Python module): the agent anticipated all three from the manifests and tests and passed without a repair attempt. That is good for users, but it means the repair loop itself has been exercised on one seeded runtime-version break so far (now re-run in the public project). The three real repos also passed first try, so they say nothing about the loop.

Challenges

  • Custom-flow schema rules I only learned by trying (no name: in the YAML, default_route inside routes, structured output not supported).
  • Router conditions need an exact one-word answer. The model wrapped it in a sentence and the router fell to end, silently. I switched to supervisor mode with delegation by tool call.
  • Sub-agents do not receive the delegation text, only their declared inputs, so the plan could not be handed from one agent to another. I merged recon and writing into one setup agent.
  • A permission-limited hackathon workspace (Developer role): no project import, no default-branch change, no access tokens through the UI, per-project flow enabling.

What I learned

Treat everything the agent reads (logs, READMEs) as untrusted data; keep the agent's edit scope tiny; make the verdict honest about what was inferred versus documented (an early version said "runs as documented" when every step had been inferred, so I changed it).

What's next

A deploy stage (for example a manual Cloud Run job); a drift watch that triggers when the README changes; testing on more messy, older repositories; a second-person install test.

Built with

GitLab Duo Agent Platform (custom flow, supervisor mode), GitLab CI/CD, Python (checker), YAML, Bash.

Limitations

  • Needs GitLab Duo Agent Platform and credits; the repository must live on GitLab.
  • Node and Python only; setup must work on shared runners.
  • One flow enablement per project; agent sessions occasionally drop and a retry recovers.
  • Auto-retry notes are posted as the token's owner and the token must be scoped carefully; do not enable it on code you do not trust.

How judges can check it

Open the live site, https://showcase-a87272.gitlab.io/: it replays real runs step by step (verdict, steps, the failing log line, each agent's reasoning), and you can paste any public merge request to render it live. The five featured runs come from the public cold-start-demo project, so each replay links to the real merge request, pipelines and commits, and you can open them yourself. Three further seeded recordings (missing build step, missing system package, a removed Python module) come from a private test project and have no public links. The flow itself cannot be triggered without a project role, so verify from the repository: README.md (install and usage), flow/cold-start.flow.yml, docs/EVIDENCE.md (the agents' real notes with timestamps), docs/SPIKES.md (what I tried and learned), and the pipeline history of this repo.

The CI/CD pipeline history showing the automation running is in the public demo project, cold-start-demo: 10 coldstart/verify pipelines, 8 green and 2 red (the seeded Node 14 failure and the needs-a-human stop), created by the flow's service account. This repository's own pipeline lints and tests the flow and the Console and publishes the site. The README's "Where to see it running" section links each run.

Licenses and third-party material

All code, the agent flow YAML and the docs are MIT-licensed (LICENSE). The Console's fonts (Fraunces, Instrument Sans, JetBrains Mono) are self-hosted under the SIL Open Font License 1.1. The three real repositories used as test subjects were copied in unmodified with their own licenses: expressjs/cors (MIT), pallets/itsdangerous (BSD-3-Clause) and kennethreitz/records (ISC). The demo video's background music is "Transmission" by SRG774 (Dark Sci-Fi Audio Pack, CC0 1.0), from OpenGameArt: https://opengameart.org/content/dark-sci-fi-audio-pack

Feedback on GitLab Duo Agent Platform

What worked well:

  • The mention trigger starts a session in about 10 seconds, and the composite identity (flow account plus the triggering user) is a sensible safety model.
  • Supervisor mode with delegate_task is robust where router conditions are not.
  • The checkpoints API (/ai/duo_workflows/workflows/<id>/checkpoints) exposes the agents' full conversations, which made debugging possible.

Friction points:

  • Docs and catalog disagree. The docs list name/description as optional top-level keys, but the catalog rejects them. A docs example puts default_route inside routes while the prose puts it beside it; the catalog accepts only the first. Errors appear one at a time, after pasting.
  • Silent failure in routers. When the model's answer did not exactly match a route, the flow fell to default_route and ended with no message. A visible "default route taken" event would have saved hours.
  • Sub-agents do not see the delegation text (observed), although the docs say the history is seeded with it. This forced a design change.
  • CI job traces hide the agents' messages. Only the checkpoints API shows what an agent said or which tool it called; this is not documented where you would look first. A session view with the tool calls would help.
  • Sessions sometimes end as failed (dropped) with no cause shown (it happened once in my test runs).
  • IDE-only tools (read_file, find_files, list_dir) appear in docs alongside web tools; using one in a web flow is a quiet failure risk. The checker now blocks it.
  • Introspection is restricted, so I could not discover how to enable a flow programmatically; enabling is per project in the UI, and there is no group-level control for a Developer.
  • Structured output is not supported in custom flows, which makes routing on a model's decision fragile.
Share this project:

Updates

Submission history