Prompt to Prod

Six GitLab Duo flows run an AI app builder from issue to production, with a person approving every step and the carbon of every generated app measured.

Comment

Inspiration

Code generation is solved; the work after it is not. LlamaCoder is a popular open-source app that turns a prompt into a working React app. Like most side projects, everything after the code was manual: no pipeline, no review, no release notes, no monitoring, and no idea what each generated app costs the planet. Each generation is one large model call, so the carbon is real and grows with every user.

We wanted to see how much of that work GitLab Duo agents could take over, while a person still makes every decision.

What it does

Prompt to Prod is our fork of LlamaCoder. You describe an app, it streams the code from a model on Amazon Bedrock and runs it in a sandboxed preview in the browser. The model picker shows the estimated carbon of one app for each model.

Around it, each stage of the life cycle is a GitLab Duo flow. A flow starts only from a human action and never takes the next step itself.

Stage What starts it Flow What it does Human gate
Plan Issue created Triage Reads the issue and the code, posts a plan with files, risk, test plan and carbon impact, labels the issue Decides whether to build it
Create A mention Developer (foundational) Builds the plan, runs lint and tests, opens a draft merge request Reviews the merge request
Verify, Secure, Govern Assigned as reviewer Review Checks the diff, the pipeline, SAST and secret detection findings, and a policy file (allowed models, one region, protected paths, token limits) Approves; two approvals for protected paths
Release Approval, or a mention Release Commits the version bump and changelog, posts release notes with a rollback plan Merges
Package, Configure, Deploy The merge GitLab CI Builds the container image, migrates the database, deploys to Lambda, smoke tests every model, publishes the GitLab release Started by the merge
Monitor Pipeline on main finishes Monitor Posts a health and carbon report to a health log issue; on a regression opens an incident with a rollback proposal Decides whether to roll back
Improve A mention Carbon Reads the latest carbon report and the code, opens one issue for the change with the best carbon saving per hour of work, with the arithmetic Decides whether to build it

That covers all nine stages: plan, create, verify, package, secure, release, configure (Terraform and the policy file, both reviewed like code), monitor and govern.

The pass in the video, recorded live

Issue #5 asked for the CO2e cost next to each model in the picker. Triage posted a plan about a minute later, and Developer opened merge request !5 three minutes after that. Review found that the estimate used the 13,000-token ceiling instead of a typical answer, so every number was 3.5 times too high. A person asked Developer to fix it, and it pushed the fix. A person signed off with a comment, Release committed version 0.3.0 with the changelog and a rollback plan, a person merged, and the pipeline deployed it and published release v0.3.0. About one hour from issue to production.

Earlier passes, all visible in the project

  1. Issue #1: Triage planned structured logging for every model call.
  2. Merge request !1: Developer built it. Review flagged a change to a protected path and a missing test; a person fixed both. Release committed version 0.2.0 and caught a stale line in the description.
  3. The merge deployed it and published release v0.2.0. Monitor posted the first report with real token counts.
  4. Carbon read that report and opened #4: halve the Lambda memory, for about 16% less CO2e per app.
  5. Developer built it in !2. Review asked for proof that half the CPU would not push generations towards the timeout. We ran the same prompts at both sizes, found a real bug on the way (a generation cut off at the 270 s limit was logged as a success and shown as finished), and fixed it in !3. The 16% estimate held up.

Monitor also got one wrong. It blamed a deploy for an outage that had ended before it and proposed a revert (#3). A person found the real cause, closed the incident without reverting, and fixed the report so errors that stopped before a deploy are no longer blamed on it. This is why we chose Assisted: the agents do the reading and the writing, and a person decides.

How we built it

  • Flows as code. The five custom flows live in .gitlab/duo/flows/. A script (scripts/duo/sync-flows.ts) pushes them to the GitLab AI Catalog, enables them on the project and sets their triggers and filters, so the repository is the source of truth and flow changes go through review like any other change.
  • Policy as code. .gitlab/duo/policy.yml lists allowed models, the one allowed region, protected paths and token limits. The Review flow checks every merge request against it.
  • Deploy. GitLab CI builds a container image, pushes it to ECR, runs the database migration and deploys to AWS Lambda in eu-north-1 with response streaming. AWS access uses GitLab OIDC; no AWS keys are stored in GitLab. Infrastructure is Terraform in infra/, secrets are in SSM.
  • Models. We moved every model call from Together AI in the US to Amazon Bedrock in Stockholm, and blocked cross-region inference so calls cannot leave the region.
  • Observability. Every model call writes one JSON log line (feature, model, tokens, latency, status). The observe job reads those lines and the Lambda metrics from CloudWatch after each deploy and writes the report the Monitor flow posts.
  • Carbon. We report Software Carbon Intensity (ISO/IEC 21031) with one generated app as the functional unit:

$$\text{SCI} = \frac{E \times I + M}{R}, \quad R = \text{one generated app}$$

The result: about 47 mg CO2e per app, against 231 mg for the same work on upstream's US setup, about 80% less. The details are in the Environmentally Impactful section.

Everything above was added on or after 5 October 2026, on top of upstream commit 75a3a78.

Challenges we ran into

  • GitLab does not let you approve your own merge request, so in a one-person project the "approved" trigger for the Release flow never fires. The human gate became a comment that mentions the flow; in a team, a teammate's approval starts it.
  • CloudWatch Logs Insights treats input and output as reserved words, which broke the token query with an unrelated error message.
  • The AWS SDK ends a model stream quietly when it is aborted, so a generation stopped by our own time limit looked like a success. We only found it because the carbon numbers did not add up.
  • Our own first before and after comparison was wrong: replaying an old window priced it at the current memory size. We caught it, fixed the script and corrected the numbers in public on the issue.

Accomplishments that we're proud of

  • Complete passes from issue to production, and from a health report back to a new carbon issue, with every step visible in the project's issues, merge requests and pipelines.
  • The agents found real problems: a protected-path change, a missing test, a stale line in a merge request description, a carbon estimate built on the 13,000-token ceiling (3.5 times too high), and a proposal whose saving we then measured.
  • When an agent was wrong, the design made that cheap: it proposed, a person said no, and the fix went into the next report.

What we learned

  • Agents are good at reading everything and writing it down; people are still better at deciding. Most of the value came from the comments, not from the agents changing code.
  • Measure before you claim. Two of our own carbon numbers were wrong until we checked them against the raw logs, and a third was caught by the Review flow.
  • At low traffic the agents cost more carbon than the app. How often the agents run is a real lever.

What's next for Prompt to Prod

  • Let Monitor skip pipelines that change nothing in production, which would cut the biggest share of agent carbon.
  • A Supervised mode for low-risk changes (docs, copy), where the flows chain on their own and a person approves only the production deploy.
  • Track SCI per app over time on a chart in the health log.

Built With

Share this project:

Updates

Submission history