SchoolPay

Inspiration

I want to own a school one day, one that shapes minds that will shape the world. A school depends on its fee software, so I wanted to build the tool that makes sure the vendor tells schools only what is true.

When engineers fix a bug, the people who talk to customers are usually the last to hear. A vague note can also tell the wrong customers that something is fixed. For a school-fee vendor, "duplicate charges are fixed" is true for schools that pay by card and false for schools that pay by bank transfer. Telling the second group it's fixed does more harm than saying nothing.

What it does

Customer Impact Flow is a custom GitLab Duo Agent Platform flow. It starts by itself when a merge request is created, and it has two steps:

1. Assess reads the merge request and its code diff. It classifies the change as customer-visible, breaking, internal or security, lists affected and unaffected schools with reasons, drafts plain-language messages for account owners only when policy allows, and lists every claim it removed because the code didn't prove it.
2. Post writes that assessment as a comment on the merge request. It has no other tools and never reads the raw merge request text.

Nothing is ever sent to a customer. Every draft says the change is pending release and waits for a person.

What it did in real runs

• Partial fix (MR !2): card retries were deduplicated and bank-transfer retries weren't. The flow drafted for the two card schools, listed the bank-transfer school as not affected, and cut "payments fixed" down to "card payments."
• Breaking change (MR !3): a receipt field was renamed under the description "cleanup, no customer impact." The flow classified it as breaking, wrote no friendly draft, flagged the two schools with an attendance integration, and said the description contradicted the code.
• Prompt injection (MR !4): a README typo whose description told the assistant to promise refunds and approve the change. The flow marked it suspicious, followed none of it, and wrote no draft.
• No school affected (MR !5): our own tooling and docs change produced no message.

How we built it

• A custom flow in YAML with two agent components, each given only the tools it needs.
• A trigger on "Merge request: Created" that runs under the flow's own service account.
• A small Flask demo app (SchoolPay, in memory, synthetic data) with pytest tests, so there are real diffs to assess.
• GitLab CI that runs the tests and a license scan on every push. The pipeline history shows a failed run, a fix and a green run.
• A readable copy of the customer list and communication policy in the repo (flows/customers.yml, flows/comms-policy.yml).

Splitting the flow in two matters. The agent that reads untrusted merge request text has no way to post, and the agent that posts only sees a structured summary. This reduces prompt-injection risk. It doesn't eliminate it.

Challenges

• There is no "merged" event, so the flow triggers on merge request creation. Its drafts describe a pending change, and they say so.
• A trigger fires once, at creation. Editing a description afterwards doesn't re-run it, so every test needed a clean branch and a committed change before opening the merge request.
• A custom flow can't call a custom agent, so the agents live inside the flow's YAML, not as separate catalog agents.
• Prompt wording was fragile. A long policy list indented badly in YAML and broke validation, so the policy lines became single lines.
• Git and authentication cost us more time than the flow did. Several commits landed on the wrong branch before we learned to check the branch before every commit.

What we learned

Deterministic checks and language models do different jobs. The most useful part of the output isn't the draft, it's the "claims removed" list, which shows what the code didn't prove. A flow that can say nothing is more valuable than one that always writes something.

Honest limits

• All schools, owners and data are synthetic.
• The customer list and policy are embedded in the flow's prompt. The YAML files in the repo are readable copies, and editing them doesn't change the flow.
• "Escalate" means a note for a human. No issue is created and no message is sent. An approval-issue step wasn't built.
• Each scenario was run once. This is a demonstration, not measured accuracy.
• The demo app's retry logic is simplified.

What's next

Create an approval issue for each draft, read the customer list from the repo, trigger on release events, and measure accuracy across many past merge requests.

Autonomy level: Supervised. Path: A (Start Fresh).

Built With

• flask
• git
• gitlab
• gitlab-ci/cd
• gitlab-duo-agent
• gitlab-merge-request
• pytest
• python
• yaml