Second Look: a scam-message helper shipped by Duo agents

A live app that helps people in India take a second look at a suspicious message, now released through GitLab CI/CD and Duo agents that block any change that could make it call a scam "safe".

Comment

Inspiration

In Thane, a 63-year-old man got a message saying his bank reward points were about to expire. It had a link and even his real customer ID, so it looked genuine. He followed it and entered an OTP, and nearly ₹50,000 left his account within minutes. In Ahmedabad, an 80-year-old retired government employee who lives alone lost his entire life savings of ₹17 lakh, built over 38 years of service, in about 30 seconds.

Neither of them needed a security expert. They needed someone to say "wait, take a second look" before they acted, and to tell them what to do next. That is what I built. Second Look lets anyone paste a suspicious SMS, WhatsApp message or email and get calm, official guidance: which warning signs it shows, what to do next, and when to call the national cybercrime helpline, 1930.

The app worked, but how I released it was risky. I tested on my laptop when I remembered, deployed by pressing a button on two Render services, and never re-checked the AI after a change. For this tool, one careless change can cost someone money. The worst would be a change that lets the AI agent tell a frightened person a scam message is "safe". That is the problem I took to GitLab: life after code for a safety tool.

What it does

The app (live at https://second-look-ai-1.onrender.com):

  • Finds warning signs in a pasted SMS, WhatsApp message or email and quotes the exact words that triggered each one.
  • Gives guidance only from verified official sources, plus next steps for "I clicked", "I shared my OTP" or "I paid", with a Call 1930 button.
  • Has an optional AI agent (Gemini, falling back to Groq) that investigates step by step using allow listed tools, asks the person one question, and writes a report they can read out to the helpline, in English or Hindi.
  • Never gives a verdict. Code-level checks reject any agent summary that says or implies a message is safe, genuine or a scam.

The GitLab automation (what I built for this hackathon):

  • Tests: before, on my laptop when I remembered. Now, more than 850 backend, ML and frontend tests run on every merge request.
  • Code review: before, none. Now, a Duo flow reviews every merge request against the project's safety rules (AGENTS.md) and posts a review.
  • Security: before, none. Now, SAST, secret detection, and Python and npm dependency audits.
  • AI quality: before, a one-off evaluation. Now, a quality gate re-measures the agent's guardrail, guidance retrieval, warning signs and the classifier, and blocks the merge if any number gets worse.
  • Deploy: before, pressing "Deploy" on two services. Now, merging to main deploys both, then waits until the live backend reports the new commit.
  • Release check: before, trying a message by hand. Now, six live checks, including a full AI agent run and a check that no verdict reached the user.
  • Monitoring: before, none. Now, the same live checks run every 6 hours on a schedule.
  • When something breaks: before, noticed by chance. Now, a Duo flow reads the failed job logs, finds the cause, and reports it on the merge request or in a production-health issue.

The proof: a real unsafe change, caught

I opened a merge request that looks harmless: "allow friendlier summaries". It deletes one pattern from the agent's safety check. 👉 https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43722820/showcase/-/merge_requests/1

  • All 121 of the agent's own unit tests still passed. Ordinary tests did not notice.
  • The quality gate failed: with the rule gone, the summary "Paying is the correct and safe thing to do." got through. That is exactly what a digital-arrest scammer wants the person to hear.
  • The Second Look safety review flow commented on its own. It named the deleted rule and the file and lines, cited the two project rules it breaks, marked it Blocking, and rejected "friendlier summaries" as a reason to remove a safety control.
  • When the pipeline failed, the pipeline triage flow commented too. It traced both failed jobs to the same deleted rule, quoted the gate's FAIL row, and proposed the fix. It also said explicitly not to lower the baseline or weaken the tests.
  • Nothing was deployed. The unsafe change never reached the live site.

The quality gate earned its place before this demo. On its very first run it found a real gap in the guardrail, the same "correct and safe thing to do" sentence, which I then fixed.

How I built it

  • App: FastAPI and Python 3.11, React 19 with TypeScript and Vite, a scikit-learn classifier trained on the UCI SMS Spam Collection, and Gemini and Groq for the agent. Hosted on Render.
  • CI/CD (.gitlab-ci.yml): stages test → quality → deploy → verify. Merge request pipelines, GitLab SAST and Secret Detection templates, JUnit reports in the merge request, an environment with a resource group for deploys, protected and masked variables for the deploy hooks, and a pipeline schedule for monitoring.
  • Quality gate (backend/evaluation/quality_gate.py):
    • It compares against a committed baseline, and each number may only get better.
    • The "no verdict gets through" rule is marked fixed, so even updating the baseline cannot relax it.
    • It checks 38 verdicts and attack outcomes. It also checks 22 real summaries written by Gemini and Groq, so the guardrail cannot "pass" by blocking everything.
  • Release tooling (ops/release.py): triggers the Render deploy hooks, polls the backend's /version until it serves the pipeline's commit, then runs the live checks.
  • GitLab Duo Agent Platform: two custom flows (.gitlab/duo/flows/).
    • They are triggered by Merge request created, Assign reviewer, Pipeline failed and Mention.
    • Their tools are read-only plus comment and issue tools. Neither can edit code, push, merge or deploy.
    • AGENTS.md gives every agent, including GitLab's own, the project's eight non-negotiable rules.

Challenges I ran into

  • Making a guardrail measurable. A pattern list is easy to weaken without noticing. Turning it into a gate with real model output meant collecting attack cases and honest summaries, and making the key rule impossible to relax.
  • Proving a deploy really happened. Render's deploy hook returns before the build finishes. I added a /version endpoint so the pipeline waits until the live backend serves the exact commit it tested.
  • Agents that report, not act. I deliberately gave the flows no write tools, so a human always decides. A safety tool should not let an AI merge or deploy its own fixes.

Accomplishments that I'm proud of

  • A real unsafe change was blocked three ways: by the gate, by the review flow and by the triage flow. All of it happened on its own, before any person looked.
  • The live site is now released only by the pipeline, and every release is checked with a real AI agent run.
  • The Duo flows were accurate. Every number and file they quoted matched the actual logs.

What I learned

Tests check that code does what you wrote. A quality gate checks that it still does what users need. For an AI feature, only the second one catches the change that matters. I also learned that AI agents are most trustworthy in a pipeline when they explain and recommend, and people decide.

What's next for Second Look

  • Hide OTPs and account numbers before any message reaches the AI.
  • Add Hinglish and native-speaker-reviewed rules for Telugu, Urdu and Bengali, each guarded by the same gate.
  • Let the triage flow open a draft fix merge request for a human to approve.

Built With

Share this project:

Updates

Submission history