AniGuard

• 

  Cover image

About the project

What inspired me

My Anime Hit Predictor was a pile of notebooks. It trained a model, printed some numbers, and then sat there. Nobody could use it, and nothing around it ever changed.

Then I read the "Life After Code" theme and asked a simple question: what happens to a project after the code is written? For my predictor, the answer was nothing. So I gave it a life. I put GitLab Duo agents around it and let them run everything after the code: review, testing, security, release, config changes and monitoring.

The best part is the threshold. A producer with a small budget wants fewer picks that are almost always right. A producer with a big budget wants to catch every possible hit. Same model, different cut-off. Now the producer files an issue with a budget and a priority, goes away, and comes back to find production already changed and the issue closed with a summary.

What I learned

Duo is better than I expected, if you give it a narrow job.

• Flows can start themselves. A "work item created" trigger meant the agent ran the moment an issue was filed, with no mention needed.
• Custom agents plus flows let you split work into steps (read the request, then apply the change), which behaves much better than one big prompt.
• Duo code review caught real mistakes. It blocked one of my own MRs because the README still showed an old threshold.
• Agents only know what their tools let them see. Mine could not list CI jobs, so I stopped asking it for metrics and let CI measure them.

How to use AI agents better:

• Give each agent one small task and a strict output format.
• Tell it when to do nothing. A NO_ACTION rule saved me from a flow that kept acting on unrelated issues.
• Put the hard limits in code, not in the prompt. The agent can only pick a threshold between 0.50 and 0.90.
• Keep the gates (tests, scans, approvals) outside the agent so it cannot skip them.

The precision and recall trade-off is a business decision. On my validation set (2022):

$$\text{precision} = \frac{TP}{TP + FP} \qquad \text{recall} = \frac{TP}{TP + FN}$$

High precision means few false alarms. High recall means few misses. Industries pick differently: fraud teams, spam filters and hiring screens usually want precision, while medical screening and safety checks want recall. Today that choice is usually buried in code and changed by an engineer. This project shows it can be a governed setting that a non-engineer changes by filing an issue, with every safety check still running.

How I built it

1. Model: I kept my LightGBM model (trained up to 2021, validated on 2022, tested on 2023 and later) and split the notebooks into a model/ package. I also added a four-tier model (Flop, Semi-hit, Hit, Breakout hit) and an AniList lookup so the agent can predict any anime by name.
2. Service: a FastAPI app in Docker, deployed on Render. The threshold is read from config/threshold.json, so changing it is a normal commit.
3. Agents: three Duo flows. One handles threshold requests, one guards every MR, one answers "will this anime be a hit?" issues.
4. Pipeline: CI runs the leakage check, SAST, secret detection and dependency scanning. After that, an auto-merge job merges the MR, a deploy job triggers Render, and a verify job waits until the live /health shows the new threshold before it comments on the issue and closes it.
5. Guardrails: a custom leakage check blocks features that would let the model cheat, plus two security policies (always scan, and block merges with new Critical or High findings).
6. Monitoring: a health probe every 6 hours that opens an incident issue if something fails.

Challenges

• Broken pipelines: my first auto-merge job failed YAML validation because of a curl command. Wrapping it in a - | block fixed it. Dependency scanning also failed until I added a test stage.
• Local errors: the API crashed on startup with KeyError: 226 while loading the model file. Fixing that took a while.
• Duo blocking my own merges: Duo requested changes on a threshold MR because the README had a stale number. I removed hardcoded numbers from the README so it stays correct.
• A slow deploy: one run failed because Render took about 4 minutes to rebuild, longer than my verify job waited. The incident job opened an incident issue, as designed. After Render finished, I retried the job and it passed.
• Making the agent behave: early on the flow only read diffs and reported "no relevant change". I rewrote the prompts, narrowed the inputs and added the NO_ACTION rule. I also had to disable and re-enable the flow every time I changed its triggers.
• The leak demo: my first try failed for the wrong reason because I added duplicate lines by accident. I redid it with a clean change, and the guard blocked it properly.
• Limits: no credit card for Duo setup (GitLab waived it), no free Docker hosting on Hugging Face (I moved to Render), and few CI minutes (I cut the health probe from every 30 minutes to every 6 hours).

Built With

• anilist-api
• codecarbon
• devsecops
• docker
• duo-agent-platform
• fastapi
• git
• gitlab
• gitlab-ci-cd
• gitlab-duo
• lightgbm
• machine-learning
• markdown
• matplotlib
• mlops
• numpy
• pandas
• python
• render
• scikit-learn
• semgrep
• shap
• uvicorn
• yaml