Downstream

• 

• 

• 

• 

• 

• 

Inspiration

Look at how a change reaches production today, even with autonomous flows. A user or maintainer opens an issue with steps to reproduce it. The maintainer assigns it to an agent. The agent writes the fix and opens a merge request, sometimes all in one go. A review agent or a human reviews the code, CI runs the tests, and the maintainer merges.

None of those steps does what a real user would do. Nobody walks through the app the way a user does after the change lands. That's why users still end up on X complaining that something broke or got slow after a deploy, and why companies often find out from support tickets instead of their own pipeline. New Relic's 2026 Observability Forecast found 42% of organizations still learn about disruptions through channels like customer complaints and manual checks. Google's 2025 DORA report found about 6 in 10 teams say more than 8% of their production changes degrade service and need a hotfix or rollback. Those changes had already passed review and CI. We wanted something that tests a change the way a user would use it, before it ships.

What it does

Downstream turns any merge request into real user journeys, catching what users would hit that code review and tests missed.

One mention on a merge request starts it. Downstream works out which routes the change touches, including routes that share code with the changed ones, and writes up to 20 journeys covering them. It shows those journeys to the reviewer in plain English first. Then it runs them twice: Does the journey break? It walks the new feature the way a user would and flags any step that fails.

Did the experience get worse? It runs the same journeys on the code without the change and compares step failures, CPU per journey, memory, bytes written, syscalls and threads. Latency is reported too, as advisory.

So it doesn't just catch crashes. It catches the change where the new feature works, but an existing feature quietly gets slower or heavier for users. That usually happens when a change unexpectedly affects another part of the app.

False positives are filtered by design. The old build runs twice in the same session, before and after the new one, so a change is only blamed for what exceeds the gap between those two runs, plus a per-metric tolerance calibrated from runs on identical code.

Once a finding is confirmed, Downstream opens one issue per cause. Each issue has repro steps, the likely cause and numbered fix options. Replying fix it commits the chosen fix on a new branch, opens a merge request for the requester and re-measures it. Downstream never merges; a human always does.

It works on backend and frontend changes. For pages, it checks bundle growth (and which asset caused it), missing assets, DOM growth, new accessibility and markup errors, and JavaScript errors.

How we built it

Downstream is a GitLab Duo Agent Platform flow, published in the AI Catalog, with a mention trigger. Any project can enable it.

The core is a measurement engine we wrote in Go, using only the standard library. The flow calls fixed engine commands (scan, validate, run, report, issues and more). The model writes the journeys and the sentences in the report. The engine produces every number. The model never writes shell commands for the engine and never writes a figure.

Scan: a Go AST scan maps routes to handlers and follows calls to find routes that share code with the change. The engine rejects any journey set that leaves an affected route uncovered.

Run: inside the Duo sandbox, the engine builds both versions of the app from git and starts each on 127.0.0.1. k6 runs the journeys on the old build, then the new build, then the old build again. This is user-activity simulation with a few virtual users for tens of seconds, not load testing.

Measure: process counters are read from /proc for each build, and each metric's tolerance was calibrated from runs on identical code.

Frontend: pages are rendered in happy-dom with the app's own JavaScript, plus axe-core and html-validate, so there's no browser to install or start.

Memory on Google Cloud: a sandbox forgets everything after each run, so ten merge requests that each add 8% CPU would all pass. A memory service on Cloud Run with Firestore keeps a numbers-only record of each run. Every report gets a Trend section that compounds changes across merged merge requests and flags creep.

Evidence: every claim in the repo links to a committed log, and make verify fails if any evidence file has changed since it was recorded.

Challenges we ran into

Getting the target application to build and run inside the sandbox. The sandbox has a read-only home directory, no TMPDIR, blocked Unix sockets by default, a network allowlist and a 200-second limit per command. Each had to be worked around before a single journey could run.

Older language versions in the sandbox. It ships Go 1.22, and our demo app used a package that needs Go 1.24 or newer. We rewrote it to build on Go 1.22 without losing functionality.

Frontend changes. Backend changes were relatively straightforward because journeys are API calls. Frontend changes needed a completely different way of simulating what a user sees, and it had to stay fast and cheap enough to run in the sandbox on every merge request.

Noise. On identical code, two runs differed by 32.5% in step latency at the median, which would have flagged clean changes constantly. That's why verdicts rest on process counters measured against two baselines, not raw timing.

Accomplishments that we're proud of:

Frontend journeys without a browser. No Playwright download, no headless Chrome, and none of the slowness that comes with them. It caught every planted issue type on real React and Svelte (Vite) builds and passed a clean change on both.

Catching what users actually feel. These are the performance problems that get dismissed as small until people complain publicly, not just outright failures. In one live run, a new feature made existing journeys use about 31 times more CPU (3.09 ms to 94.96 ms per journey), and Downstream caught it before merge.

It works end to end on GitLab. A mention runs the whole flow in the Duo sandbox: building both versions, writing journeys, measuring, reporting and opening issues. fix it opened a merge request that cut CPU per journey by 70% and re-measured it to prove it.

Clean changes stay clean. A clean change produced zero false regressions in all 3 live flow runs under the current rule, and in 22 of 22 CI runs under an earlier rule.

What we learned

We learned a lot about GitLab Duo Agent Platform flows: triggers, the sandbox, agent config, and how a flow can open issues, comment, commit and create merge requests. We also learned how much of the work is fitting into the sandbox you're given.

The bigger takeaway was GitLab's native AI integration. Having the review agent, the developer agent and custom flows built into the same place where merge requests and issues live felt more natural than anything we'd used before. It's something we want to keep building on.

What's next for Downstream

Scale beyond 20 journeys. The cap exists because of sandbox capacity. Most merge requests don't need more, but large changes might.

More languages and frameworks. Backend route mapping currently covers Go net/http.

Become the step teams run before production. Once review, CI and the usual checks have passed, Downstream should be the last step before deploy, for small teams and enterprises alike, showing the actual impact of a change on users before they find it themselves.

Built With

• gcp
• gitlab
• go