Ghost Customers

• 

  issue activity on gitlab

Inspiration

Most teams find out a release hurt someone from a support ticket days later, if the person bothers to write one at all. A guest who can't tap RSVP on her phone, a blind user whose screen reader can't name a form field, or an organizer whose Excel export won't import usually just leaves. Code generation is solved, but nobody checks the result as a customer before and after it ships.

So, I asked: what if every build came with its own customers, who show up the moment it's live, notice what's broken, complain in their own words, and only let a fix through once it works for them?

What it does

Ghost Customers gives a project a panel of three synthetic customers:

On every pipeline they walk the app, and then the loop runs with no human in it:

1. Walk. A CI job sends each persona through the real container image with Playwright and prints a JSON report: passed or blocked, the steps, measurements and a screenshot.
2. Report. A GitLab Duo custom flow reads the report and opens an issue in the persona's voice, with steps, evidence, a screenshot link, "done when" criteria and rules for the fix.
3. Fix. GitLab Duo Developer is assigned and opens a merge request with the fix and a regression test.
4. Verify. The merge request pipeline runs tests, SAST, secret detection, a deterministic merge gate and the persona walk against the preview build. The persona reads its own result and either approves the exact commit in a note ("I tried this preview at commit 8aa196ca and it finally works") or sends it back to draft explaining what still fails.
5. Merge. Only a fix the persona approved, at that exact commit, with green tests, merge gate and walk, gets merged.
6. Recheck. After the merge, the persona walks production again and either confirms the fix (verified-live) or reopens the issue.

In the first real run, all three personas found their problem, three fixes were written, verified, merged and confirmed on production, and the final walk is green. When Sam's fix merged first, Priya's conflicted with it. The loop asked Duo Developer to rebase, it pushed a clean commit, and Priya re-verified the new commit before it merged. No human touched any of it.

How I built it

• The app: Harvest Supper Club, a small Flask RSVP site written for this project, shipped in a non-root gunicorn container. It has three deliberate defects, so the first walk finds something real: a fixed 960px layout, an email field with only placeholder text, and a CSV import that rejects Excel's UTF-8 byte-order mark.
• The ghost walk (personas/): deterministic Playwright journeys that emulate a phone on throttled 4G, keyboard-only navigation with accessible-name checks, and a byte-exact Excel CSV upload. The personas test general things any customer hits, not the specific bugs, and each journey was verified to pass once its defect was properly fixed.
• The Ghost Customers flow (.gitlab/duo/flows/ghost-customers.yaml): a GitLab Duo Agent Platform custom flow on a Pipeline events (Passed) trigger. It reads walk reports with get_job_logs, reads each persona's voice from project skills (skills/<persona>/SKILL.md), opens and closes issues, reviews merge request pipelines with gitlab_api_get, and approves or rejects fixes as the persona.
• GitLab Duo Developer (foundational flow) writes every fix. Fix CI/CD Pipeline is on Pipeline events (Failed).
• The merge gate (scripts/gate.py): plain code, not an agent. A fix may only change app/ and tests/. Persona journeys, skills, the flow, CI, the gate, dependencies and the license are off-limits, so no agent can "fix" a bug by editing the test that found it.
• ghost-dispatch (scripts/dispatch.py): a small CI job on main that carries out the personas' decisions. It assigns Duo Developer to new issues, merges only persona-approved commits, asks Duo Developer to rebase conflicting fixes, and passes on "please take another look" requests.
• GitLab CI/CD: unit tests, SAST, secret detection, a container build to the GitLab registry tagged with the commit SHA, the walk, and dispatch. A pipeline schedule on main drives the loop.

Challenges I ran into

The hardest part was a platform rule I didn't know about: a flow cannot wake another flow. GitLab triggers only fire on human actions.

• The flow's attempt to assign Duo Developer was silently dropped. A bot mention can't trigger it either.
• Duo Developer's own merge request pipelines don't start the persona flow, because a bot started them.
• Calling start_flow from the flow failed with "Insufficient permissions to create a new pipeline", and merging from it failed with 403 insufficient_scope.

The answer became the design: personas decide, ghost-dispatch acts. The agent does all the judging: reading evidence, writing as the persona, approving or rejecting a specific commit. A small, auditable script with a protected token carries out only what a persona approved. Its assignments count as human, so they fire the Developer trigger, and its merges start a new pipeline that wakes the personas to recheck production.

Real runs found more edge cases, and each became a rule:

• GitLab auto-closes an issue when its fix merges, before the persona has rechecked. Now the persona rechecks closed issues too and reopens them if production still fails.
• A walk of a commit from just before a merge filed a duplicate issue (#4). Now walks that predate a merge are ignored.
• Two fixes conflicted in the same test file. Now the dispatcher asks Duo Developer to rebase, once per commit.
• GitLab's lazily rechecked merge status kept reporting "conflict" after the rebase. Now only has_conflicts is trusted.

Accomplishments that I'm proud of

• A complete loop, from finding to issue, fix, verification, merge and production check, that ran end to end on real defects without anyone clicking anything.
• Merge authority sits with the customer: a fix ships only when the persona confirms that exact commit works for them.
• Agents never grade their own homework: the merge gate is deterministic, and the tests that find bugs are off-limits to the agents that fix them.
• Issues and reviews read like real customers, which makes them pleasant to read and easy to act on.

What I learned

• Agentic automation needs a clear line between judgment (agents) and authority (deterministic code with narrow permissions). That line made the system safer and the platform's limits easy to work with.
• GitLab Duo Agent Platform's building blocks (custom flows, triggers, project skills, foundational flows, scoped tokens) compose well once you design around the human-trigger rule instead of fighting it.
• Accessibility and real-device checks are perfect jobs for agents: concrete, repeatable, and usually skipped.

What's next for Ghost Customers

• Walk a real production deployment (for example Cloud Run) after every release, not just the CI-built image.
• Let teams define personas from their analytics: their real devices, assistive tech and data formats.
• More journeys per persona, with trend tracking across releases.

Built With

• docker
• flask
• gitlab
• gitlab-ci
• gitlab-container-registry
• gitlab-duo
• gitlab-duo-agent-platform
• gitlab-sast
• gunicorn
• playwright
• pytest
• python
• secret-detection