SafeFlow

Inspiration

As a freelance web developer, I kept running into the same problem: AI security tools that sound confident but show no evidence. Hallucinated line numbers, invented vulnerabilities, "trust me" findings. I wanted to build the opposite: an AI reviewer that behaves like a good security engineer — it shows you exactly what it read, admits what it did not check, and never touches your repository without a human saying yes.

What it does

SafeFlow is an evidence-first DevSecOps risk reviewer for GitLab, with two layers:

• A deterministic Python engine (no AI): rules that detect CI/CD risks like security gates with allow_failure: true, findings with stable rule IDs, severity, evidence confidence, JSON and Markdown reports, and a CLI.
• A read-only GitLab Duo custom flow ("SafeFlow Review") built on the GitLab Duo Agent Platform: it explains risks from real repository files, proposes minimal remediation — and pauses at a human approval gate. It has no file-editing, commit, or merge-request tools at all.

The core idea is an evidence-source taxonomy: every claim the AI makes must be labeled as repository-file (with the exact file path and line range read), issue-description, synthetic-fixture, or unchecked. If it could not verify something, it must say so — never guess.

How I built it

I built the deterministic engine first, so every AI behavior could be tested against a ground truth: rules, a findings model, reporters with secret redaction, a CLI, and 25 unit tests including a regression scenario that verifies a finding disappears after the fix. All tests run on GitLab CI on every push.

Then the Duo flow: a read-only toolset (read_file, find_files, list_dir, get_issue, list_issue_notes — nothing that writes), an approval HumanInputComponent with approve/modify/reject routes, and a system prompt enforcing the evidence taxonomy. Config guardrail tests assert the flow file keeps its read-only shape and evidence requirements.

Finally, I runtime-tested it for real: two sessions triggered from GitLab issues by mentioning the flow's service account, verified through the GitLab API and session logs.

Challenges I ran into

• No REST API exists for Duo flows or sessions. I verified runtime behavior by combining the issue notes API, the todos API, and reading session pages — and only documented what I could prove.
• The flow cannot confirm which ref its file tools read (access to .git/HEAD is blocked). Instead of hiding this, I made it a requirement: the flow must mark that claim unchecked. Watching the AI honestly flag its own blind spot was the best moment of the project.
• Proving "read-only" is hard to fake. I ran the rejection path end to end: the maintainer rejected the proposal, then I verified via the GitLab API that zero commits, pushes, or branch changes occurred at any point during the session.

Accomplishments I'm proud of

• Session 9412534: the flow located and read the real .gitlab-ci.yml (lines 1–15), cited line ranges for every claim, labeled each with its evidence source, and stopped at the approval gate.
• The verified rejection path: rejected proposal → zero repository writes, confirmed by API. Human control isn't a UI promise; it's a tested behavior.

What I learned

Evidence-first prompting changes AI behavior more than model choice. Forcing the model to label every claim's source — and to say "unchecked" instead of guessing — produced reviews I would actually trust. I also learned to verify AI runtime claims with APIs before writing a single line of documentation.

What's next for SafeFlow

More detection rules, and connecting the heuristic checks to real scanner outputs — today's rules are honestly documented as text heuristics, not a production scanner.

Built With

• agent
• agents
• ai
• api
• ci/cd
• devsecops
• duo
• gitlab
• human-in-the-loop
• platform
• python
• rest
• security
• testing
• unit
• yaml