Downstream

Path A: Start Fresh · Supervised autonomy

Inspiration

Analytics engineers don't struggle to write SQL. They struggle with what happens after: a renamed column silently breaks revenue reporting, nobody writes tests for the new model, a raw email lands in a shared table, every change rebuilds the whole warehouse, and the dashboard is wrong for a weekend before anyone notices. Most DevSecOps automation is built for web services. Data teams get little of it, so I built Downstream for them.

What it does

One person opens an issue. A single GitLab Duo flow then carries the change through the whole post-code lifecycle:

• Plan: the triage agent writes acceptance criteria and a data-test plan on the issue.
• Create: the developer agent implements the dbt change and opens a merge request.
• Verify: the lineage reviewer diffs the dbt manifest against main's, finds every downstream model and dashboard tile the change reaches, blocks contract breaks (a removed column still used downstream), fixes them, and adds missing tests.
• Secure: SAST, secret detection, dependency scanning, and a PII Sentinel that reads sampled values, not just column names.
• Package: the dashboard image and the exact tested warehouse file are versioned in GitLab's registries.
• Stage + monitor: staging is deployed and compared against the last good release; on an anomaly the pipeline rolls staging back and opens an incident.
• Release: a semantic version derived from the data contract, release notes written as a contract diff, and an evidence bundle (checks, PII, carbon).
• Deploy: one human approval (a HumanInputComponent gate) before production, then a production monitor with a rollback job.
• Govern: hard rules in AGENTS.md, every step visible in one session and one pipeline.

In a real run on our hackathon project (issue #3 → MR !2 → release v1.1.2), the lineage reviewer ran the full build (55/55 pass) and caught that the issue's own wording misstated the revenue logic: refunded orders are subtracted, not ignored. It corrected the description before merge.

How I built it

• GitLab Duo Agent Platform: a custom flow (.gitlab/duo/flows/downstream.yaml) with six AgentComponents and a HumanInputComponent approval gate routed on approve/reject; a generated Hands-off variant; an Assisted merge-request review flow; triggers on work item created, mention and assign; AGENTS.md and agent-config.yml for context.
• One orchestrator, not a chain: flow triggers only fire on human actions, so a flow can't hand off to another by merging. Downstream runs one flow whose agents drive GitLab through a small bridge (scripts/ds.py): wait for pipelines, merge, press the manual production job, read artefacts, open incidents.
• GitLab CI/CD: build, test, package, staging, release, production and monitor stages; Jobs/SAST, Jobs/Secret-Detection, Jobs/Dependency-Scanning.v2; JUnit reports; environments; Releases; Pages for dbt docs; rollback jobs.
• Data stack: dbt + DuckDB, a FastAPI dashboard, synthetic Malaysian e-commerce data.
• Tests: 44 dbt data tests and 72 unit tests, including static checks on every flow router, input and tool name.

Challenges

• Designing around the rule that flows can't trigger flows: the single-orchestrator design keeps every step visible in one session and one pipeline.
• The agents' run_command has no shell, so the whole review is one script that writes one report the agent can read, fix from and re-run.
• Found in a live run: the flow's token can't merge merge requests (403 insufficient_scope). Rather than work around the permission, the gatekeeper now asks a maintainer on the merge request to press Merge and waits, which also puts a person's eyes on every change before it reaches main.
• Also found live: after the production approval, the session errored when it resumed, so in that run I pressed the approved production job by hand. The pipeline then deployed and its production monitor passed.
• No Google Cloud billing was available, so the Cloud Run deploy step runs in dry-run mode; everything else (builds, scans, staging monitor, releases) runs for real.

Accomplishments

• The lineage reviewer catches a column rename that breaks two models before anything is built, and fixes it.
• A build that silently loses 90% of orders passes every dbt test; Downstream's monitor catches it and rolls staging back.
• While building it, the test-gap gate found a real NULL-revenue bug in my own model.

What I learned

Agents are most useful where they read context a linter can't: the lineage graph, the SQL behind a description, the data itself. Deterministic work belongs in CI, and permissions the platform withholds should be handed to a person, not routed around.

What's next

Cross-repository blast radius; carbon-aware scheduling of full refreshes into low-intensity hours; column-level lineage from compiled SQL.

Sustainability (SCI)

Every merge request is scored with the Green Software Foundation's Software Carbon Intensity (ISO/IEC 21031:2024), SCI = ((E × I) + M) / R per release, using Cloud Carbon Footprint's GCP coefficients and Google Cloud grid intensity. Merge requests rebuild only state:modified+ on top of main's warehouse instead of a full refresh: a leaf-model change scores 75% lower SCI, and a docs-only change rebuilds 0 of 8 models.

Built With

• dbt
• docker
• duckdb
• fastapi
• gitlab
• gitlab-ci
• gitlab-duo
• python
• sql