Inspiration
Agents are opening more and more merge requests, and honestly, we approve them faster than we can read them. Humans have become the bottleneck.
Security is getting worse at the same time. ultralytics shipped a crypto miner through PyPI. The Shai-Hulud worm stole developer tokens straight from npm installs. None of these looked scary in a diff. They looked like a few harmless lines. The damage only showed up once the code actually ran: it phoned home, read secrets, started processes.
Code review reads what the code says. I wanted something that checks what it does.
What it does
Side Effects runs every merge request in a sandbox, watches what the code actually does, and compares that against what the code owners said it's allowed to do.
- Declare. The repo has a small file,
.gitlab/side-effects.yml, that lists what the code may do at runtime: which hosts it can talk to, which env vars it reads, which programs it starts, whether it hooks into Python startup. It's protected by CODEOWNERS, so an MR can't quietly loosen its own rules. - Measure. For every MR, CI installs, imports and tests the old and the new version in a sandbox with no network. A small tracer records everything the project's own code does.
- Stay quiet, or block. If the change stays inside the rules, Side Effects says nothing at all. If it does something new, how loud it gets depends on what the code owners set: just note it, ask for an explanation, block the merge with a thread on the exact line, or make a human prove they understand it.
- Prove it. For risky stuff, or anything an AI agent wrote that adds network calls or processes, the reviewer gets a short quiz built from what actually ran ("which program does this start? when?"). The answers are checked against the trial, not by an LLM. Guess wrong and the merge stays blocked.
- GitLab Duo reads the fine print. A Duo flow reads the MR description and decides whether the author actually mentioned each new behaviour. If they did, it calms down. If not, it explains the gap to the reviewer.
- Release labels. Every release gets a behaviour label, kind of like a nutrition label: everything that version does, checked against the rules.
- Production. The demo app runs on Google Cloud Run with the same tracer watching. When a code path no test covered does something new, Side Effects opens an incident, points at the exact line and the MR that added it, and flags the release. A second Duo flow then suggests the fix: declare it or gate it.
The point is to bring humans back into the loop, but only when it actually matters.
How I built it
- The tracer is plain Python using
sys.addaudithook. It loads before anything else and only blames the project's own code, so pip, pytest and the standard library don't show up as noise. - GitLab CI does the work in separate jobs. The job that runs the untrusted MR code never holds a token. A second job reads the results and talks to the MR. A third one labels releases on tags.
- Two custom GitLab Duo flows: one judges MR descriptions, one triages production incidents. Their verdicts are only trusted when they come from the flow's own service account.
- Google Cloud Run hosts the bot (GitLab webhooks and production reports) and the monitored demo app. Both scale to zero, so it costs basically nothing.
Does it work on real code?
I ran the release check on the top 300 PyPI packages, comparing their last few releases (1,116 release pairs). 7 of them started doing something new at import time, all harmless. For example, matplotlib 3.11 started writing a font cache to ~/.cache, and pandas 3.1 started reading PANDAS_FUTURE. Comparing each package's latest release with itself gave zero false alarms. So in practice it stays quiet most of the time and only speaks up when something really changed.
Challenges
- Telling the project's behaviour apart from its tools. pip and pytest do a ton on their own, and it took some work to make the clean case actually silent.
- Keeping tokens away from the job that runs untrusted code.
- Duo flows strip HTML comments, so I had to pass the measured facts to the flow as a visible, collapsed block.
What's next
More languages beyond Python, and letting the rules learn: when a maintainer approves a new behaviour, Side Effects opens an MR to add it to the rules.
Log in or sign up for Devpost to join the conversation.