Decoy

A public privacy status page for your app, checked daily by a decoy user. When a promise breaks, GitLab Duo finds the cause, writes the fix and merges it.

Comment

Inspiration

Every app has a privacy policy full of promises, like "we never share your email with advertisers."

Nothing checks them after the code ships.

BetterHelp paid $7.8M (FTC, 2023) after users' emails and therapy history were used for Facebook ad targeting. GoodRx paid $1.5M (FTC, 2023) for sending health data to Facebook and Google. US hospitals sent appointment details to Meta through the Meta Pixel (The Markup, 2022). None of these were hacks. They were ordinary changes nobody checked against the promise, and many came from a tag or a setting, not a code release. We wanted a check that runs after the code is written, every day, and shows the result to the people the promise was made to.

What it does

Decoy gives your app a public privacy status page. Every day a decoy user signs up and uses the app with a fake email, name, phone number and private note, each carrying a random marker. Decoy captures every request that leaves the browser (Playwright) and the server (a recorder loaded with NODE_OPTIONS). It finds the decoy's details even when they are encoded, compressed or hashed (SHA-256, SHA-1, MD5), names the company that received them, and checks them against the promises written in decoy.rules.yml. The result goes on a status page for the app's users: where each kind of data went, when it was last checked, and every incident.

When a promise breaks, a GitLab Duo custom flow called "Decoy responder" starts from a "pipeline failed" trigger. It finds the cause, opens an incident issue, writes the fix, reruns the decoy on it, and opens a merge request. A pipeline job called merge-fix merges it once all checks pass. Nobody has to click.

It has run for real twice. Incident #1: a marketing tag added only through a setting, with no code change, sent a SHA-256 hashed email to Meta. The daily check caught it, and the flow opened a merge request that added a Content Security Policy to every page. With the same tag still on, the decoy reported "All promises kept". Incident #2: a "growth release" sent email, name and phone to PostHog and put name and email in the AI prompt, which broke 5 promises. The flow opened a merge request, merge-fix merged it, and the incident closed on its own. A later daily run while the fix was pending did not open a duplicate.

How we built it

Decoy is TypeScript and Node, with Playwright for the decoy user. The demo app, Jotter, is a small notes app with AI summaries and usage analytics. We built it on GitLab, using these features:

  • GitLab Duo Agent Platform: a custom flow (Decoy responder) with two agent steps, triage and fixer. It is started by a Pipeline events trigger set to Failed. The flow file is in the repo.
  • GitLab CI/CD: unit tests, the answer key, the privacy check, deploy, package, release, merge-fix and pages jobs.
  • Pipeline schedules: a daily scheduled pipeline, owned by a person so it can start the flow.
  • Security report: each broken promise is written as a DAST format report (CWE-359, exposure of private personal information).
  • Merge only on green: the merge-fix job sets "merge when pipeline succeeds", and acts only on decoy-fix branches written entirely by the responder that close a decoy-incident issue.
  • Issues and merge requests: every incident and fix stays on record.
  • Package registry and releases: the package job publishes Decoy to the project's npm registry on version tags, with a GitLab release.
  • GitLab Pages: builds the status page. Vercel hosts the public copy.

Challenges we ran into

  • GitLab rejected HTML comments and an unsupported field in the flow prompt. We moved the incident block into a fenced block so the flow saves and runs.
  • GitLab does not let a Duo flow's own token merge. We solved it with a guarded merge-fix job that only merges responder written fixes that close an incident.
  • GitLab Pages is members only in the hackathon workspace, so nobody outside could open the status page. The public copy is on Vercel, and the pipeline publishes to both.
  • A planted tag in our answer key raced the page navigation and gave flaky results. We moved it to page load so every planted leak is seen leaving the app.

Accomplishments that we're proud of

  • The full loop ran with nobody in it: failed check, incident, fix, passing checks, merge, closed incident (incident #2).
  • A leak that no code review could see (a tag added through a setting) was caught and fixed.
  • The answer key plants known leaks and records the right answer next to what Decoy found: 8 of 8 planted leaks caught, 0 of 3 false alarms. It is written by a script, not by hand.
  • The flow refused to edit tests and flagged a test conflict for a person instead.
  • The blind spots are published, not hidden: a salted hash, an encrypted value, and a value split in two are all missed.

What we learned

  • A scheduled pipeline can start a Duo flow, but only if a person owns the schedule.
  • A flow started by another flow or a bot never fires a trigger, which prevents loops.
  • Custom flows work best with one word answers between steps, so the triage step answers "violation" or "skip" and the router branches on it.
  • Hashing does not make personal data anonymous. A hashed email is still the email, and Decoy checks for it.
  • An honest answer key, with its misses listed, is worth more than a clean score.

What's next for decoy

  • Close the three blind spots, starting with values split across fields.
  • Add a second journey type so Decoy can check apps with more than a sign up form, such as checkout.
  • Package the privacy-check and responder flow as a one step template other GitLab projects can enable.

Built With

Share this project:

Updates

Submission history