Spektra Incident Guard

GitLab Duo agents turn every real Spektra outage into a permanent guardrail: incident-aware MR review, security scans, smoke-tested Cloud Run release, and a human approves production

Comment

Inspiration

Spektra is a personal AI agent (YouTube scout, market data, daily brief) that has run in production on Google Cloud Run since August. It shipped real production incidents, and each one was found by a person reading logs: a comma in ALLOWED_ORIGINS that broke gcloud run deploy, a deploy failure blamed on a secret when the real cause was that comma, market data silently empty because Yahoo blocks Cloud Run egress, a stock price taken from the wrong exchange, and a wrong CORS domain. Every one of them is in the public commit history of the original repo. We asked: what if every incident became a rule an agent enforces on the next change?

What it does

Spektra Incident Guard moves Spektra onto GitLab's full lifecycle (Path B) and wraps it in a Supervised post-code workflow:

  • Plan: every change starts from an issue labelled with its DevSecOps stage (stage::*).
  • Review (GitLab Duo): the Code Review Flow reads six custom rules in .gitlab/duo/mr-review-instructions.yaml, each named after a real incident. In MR !1 we replayed the comma incident: the pipeline was green and 69 tests passed, yet Duo requested changes, cited the incident rules by name, offered a one-click fix, and blocked the merge. In MR !2 Duo approved a correct change, which then went all the way to production.
  • Verify and Secure (GitLab CI): pytest with JUnit results (77 tests), ESLint, SAST (Semgrep) and secret detection on every change.
  • Release (GitLab CI + Cloud Run): the pipeline logs in to Google Cloud through Workload Identity Federation (no stored keys, only protected main of this project), builds the container and deploys a new revision with no traffic.
  • Monitor: a smoke test checks the candidate revision. /health must come from the right revision, market prices must be real (the endpoint returns HTTP 200 even when every quote failed, so the body is checked), and CORS must allow the frontend and refuse unknown origins.
  • Human gate: a person clicks promote to move production traffic. Agents and CI do the work; a human approves production.

How we built it

FastAPI + Gemini (ADK) backend on Cloud Run, Next.js frontend, Firestore and a GCS-mounted per-user memory. GitLab CI/CD with a deploy-candidate → smoke-test → manual promote pipeline, a hard guard so this repo can only ever deploy its own service, and separate data (personas_hackathon) so judges never touch real users.

Challenges

  • The Code Review Flow does not read Skills files, so the incident rules had to move to mr-review-instructions.yaml.
  • Cloud IAM changes had to be made by a human in Cloud Shell. We kept that boundary instead of automating around it.
  • Windows tooling reproduced the original comma incident twice while we built this.

What we learned

A green pipeline is not the same as a safe change. The most valuable rules came from incidents, not from best-practice lists.

What's next

Fix CI/CD Pipeline Flow on a replayed failing test, the Developer Flow turning an issue into an MR, and a post-release summary on the issue.

Built With

  • artifact-registry
  • eslint
  • fastapi
  • firestore
  • gemini
  • gitlab
  • gitlab-ci
  • gitlab-duo
  • gitlab-duo-agent-platform
  • google-adk
  • google-cloud
  • google-cloud-build
  • google-cloud-run
  • next.js
  • pytest
  • python
  • secret-manager
  • semgrep
  • tailwindcss
  • typescript
  • vercel
  • workload-identity-federation
  • youtube-data-api
Share this project:

Updates

Submission history