Blotter

When keys or people's personal data leak into your logs, Blotter cleans up, and tells you whose it was, since when, and which countries' deadlines are running.

Comment

Inspiration

Most data leaks don't need an attacker. A debug line prints the request. An error handler echoes the form back. A CI tool prints its connection string. GitLab can only hide the values you registered with it as masked variables. Everything else in a log line can be read by whoever reads the log, and on a public project that's anyone.

It happens to the best teams. GitLab itself fixed CVE-2023-3363, CVE-2025-13611 and CVE-2026-8330, all of which wrote secrets into logs. GitLab's own issue #375510 asks for secrets to be found in job logs.

What happens afterwards is worse. Ireland's regulator fined Meta €91 million, partly for not notifying and not documenting a passwords-in-plaintext breach. IBM's 2025 report puts the average time to identify and contain a breach at 241 days, while GDPR gives 72 hours from the moment you know. A small team with users in five countries faces five laws and five deadlines, with no privacy team to sort them out.

We wanted the part after the leak, the bit nobody automates, to happen by itself inside GitLab.

What it does

When something leaks into a log, Blotter:

  • Finds it. An hourly scan reads the production app's logs. Every pipeline checks its own job logs, before release and again at the end. ID numbers from 14 countries are recognised by their real checksums: CPF, Aadhaar, NINO, DNI, SSN and more, plus cards, IBANs, API keys and passwords inside connection strings.
  • Stops new copies. The leaking line is switched off in production within seconds. A job log that printed a key is erased.
  • Says whose, and since when. A confidential incident issue counts how many people were affected and from which countries, without anyone seeing a value: it uses keyed fingerprints that exist only while the scan runs. It names the merge request and the production deployment that put the line live, and lists each law's deadline and who to tell.
  • Fixes it, and learns. A GitLab Duo flow takes over. The fixer agent changes the code or the CI job and adds a regression test. If no existing rule would have caught the mistake, it writes a new Semgrep rule with its own leaky and fixed examples. A skeptic agent reviews the fix and posts READY or NOT READY.
  • Proves it. Every rule must flag the code that leaked and pass the fix before it's trusted. Guardrails then block any merge request that repeats a past leak, or that adds CI lines like echo $TOKEN or set -x.
  • Closes it properly. One person approves production. Blotter then lifts the switch, writes the breach record (what happened, effects, timeline, remedial action, as GDPR Art. 33(5) asks), and closes the incident.

A public page lists every incident Blotter has closed, with timelines and deadlines: https://blotter-prod.onrender.com/

How we built it

Blotter is Python, with no AI in the detection or the legal deadlines: those are plain code with sources. The AI is the GitLab Duo agents that write and review the fixes. It covers all nine stages:

  • Plan: confidential incident issues carrying the scope and deadlines. The due date is the earliest legal deadline.
  • Create: a custom GitLab Duo Agent Platform flow with a fixer agent, triggered by mentioning its service account on the incident. Rules for agents live in AGENTS.md.
  • Verify: a skeptic agent, unit tests (151), and a rule-proof job.
  • Package: a package job builds Blotter's container image into the GitLab registry. A CI/CD component (templates/blotter.yml) lets any project add Blotter with one include: line. Release 1.0.0 is out.
  • Secure: Semgrep guardrail rules (one of them written by the agent), a check for CI lines that print secrets, and GitLab SAST.
  • Release: staging deploys itself; production is a manual job, which is the one human approval.
  • Configure: a containment switch, set through Render's API, that the app's log filter reads.
  • Monitor: a pipeline schedule scans production logs and sweeps job logs; every pipeline checks its own job logs.
  • Govern: the breach record, GitLab's deployment history and git blame for the exposure window, and a public ledger built by CI.

There's also an MCP server, so an editor's agent can scan logs and look up deadlines without seeing values. The sample app (a marketplace onboarding sellers from many countries) runs on Render.

Proof: five real runs

All runs were on our own production app and pipelines, with synthetic test data:

  • Incidents #1, #2, #3: passwords and ID numbers in production logs, fixed by the agents (!1, !2, !4). Afterwards the guardrail blocked !3, an "AI support chat" feature that repeated the mistake.
  • Incident #6: a realistic leak. A merge request added "log why sign-ups are rejected", and FastAPI's validation errors carry the submitted form. Our rules missed it, and it reached production. Blotter counted 26 sellers from 3 countries (Brazil, India, Spain), traced the leak to the deployment that shipped it, contained it, and closed it 70 minutes after detection, with 70.8 hours to spare on the GDPR deadline. The agent wrote and proved a new rule, no-validation-errors-in-logs.
  • Incident #7: a CI job printed a database password. The job-log check caught it, held the release jobs, and the agents fixed the script (!10).

Challenges we ran into

  • Telling real leaks from noise. Our first job-log check flagged a GitLab runner line ending in token= plus a short token prefix. Spanish IBANs contain digit groups that can pass India's Aadhaar checksum, and a database URL's user@host looked like an email. Each one became a test and a fix.
  • GitLab's scoped labels. blotter::incident and blotter::closed share a scope, so closing an incident silently removed its "incident" label. That caused one duplicate incident until we found it.
  • Permissions. In the hackathon group our account has the Developer role, so GitLab refused to let Blotter erase a job log, which needs Maintainer. Blotter now says so plainly in the incident, and the README lists it as a limit.

Accomplishments that we're proud of

  • The agent recognised that our guardrail wouldn't catch a new kind of leak, wrote a rule that would, and proved it, without being told to.
  • Counting people per country without anyone, person or agent, ever seeing a value.
  • Tracing a leak back to the exact merge request and deployment, so the exposure window comes from GitLab's records instead of a guess.

What we learned

The detection is the easy part. The value is in everything after: scope, deadlines, the fix, the proof and the record. Keeping the legal deadlines in plain, sourced code, and the agents on code changes, made the whole thing something a privacy lead could trust.

What's next for Blotter

  • Readers for more log providers (Datadog, CloudWatch, Google Cloud Logging, Loki), and deleting stored copies where the provider allows it.
  • More countries' laws in regimes.yml, each with its source and review date.
  • Blotter for smart contracts: the same response for a team's own deployed contracts. Propose a pause through the team's multisig, have agents write the fix and a test, then keep the record.

Built With

  • docker
  • fastapi
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • mcp
  • python
  • render
  • semgrep
Share this project:

Updates

Submission history