mnemosyne

The AI that remembers every failure — predicts outages, annotates risky code inline, gates CI pipelines, auto-blocks critical MRs, and diagnoses failures. Powered by Gemini and gitlab on Google Cloud.

Comment

Inspiration

I've watched the same outage happen twice to the same team.

The first time, a senior engineer spent three days tracing a cascading failure back to a JWT field rename in a single merge request — one that had passed all tests, all reviews, all pipelines. The second time, fourteen months later, a new engineer on the same team made the same JWT field rename. Same cascade. Same three-day trace. Same $4.2M revenue loss.

The post-mortem was written. The incident was logged. The lesson existed — but it lived in a Confluence page nobody read before merging.

For years I thought the problem was knowledge — that if we just documented incidents better, engineers would stop repeating them. But watching teams get burned by the same patterns over and over again, I realized the problem isn't documentation. The problem is that documentation doesn't block merge buttons.

That realization changed everything.

I didn't want to build another tool that finds problems. The industry has dozens of those — linters, SAST scanners, SoC dashboards, vulnerability trackers. What I wanted was a system that acts. That sits at the moment of maximum leverage — the second before a dangerous merge goes through — and does something about it. Not a warning. Not a Confluence page. A hard stop.

That's the first thing MNEMOSYNE became: a system that converts a CRITICAL-risk MR to Draft the instant it detects a failure pattern, disabling the Merge button entirely until the risk is resolved.

Then I hit the second problem.

Predicting outages before a merge is extraordinarily valuable. But what happens after the pipeline breaks anyway? What happens when a CI job fails at 2am and the on-call engineer opens twelve browser tabs trying to figure out why security:sast is failing? Every team wastes enormous amounts of time diagnosing CI failures that have clear, retrievable root causes sitting in job logs.

MNEMOSYNE needed to be able to look backward at what just broke — not just forward at what might break. So I built a second mode: give MNEMOSYNE a failing MR URL, and it fetches the failed CI job logs, diagnoses the root cause using Gemini, and hands back a fix with an explanation. Same institutional-memory approach, applied to reactive diagnosis instead of predictive analysis.

Finally, there was the memory problem itself. Every time an analysis ran, the findings lived in RAM for the duration of the request and then disappeared. There was no accumulation, no learning, no way to answer: "Is this team's auth regression rate getting worse over time?" I wired MNEMOSYNE's Echo phase directly to Google Cloud Firestore so every analysis is persisted, queryable, and available to future analyses as real organizational memory.

The result is a system that doesn't just analyze — it remembers, blocks, diagnoses, and acts. Not another AI that surfaces findings after a deploy explodes. Something that answers the question no tool today can answer:

"If we merge this — what will break, when will it break, how much will it cost, and should we even be allowed to click Approve?"

What It Does

MNEMOSYNE is an autonomous DevSecOps intelligence platform that intercepts every merge request — via GitLab webhook or GitLab Duo Chat — and runs a full 9-phase AI analysis before approval.

The 9 Specialist Phases

Phase Agent What It Analyzes
1 Orchestrator Fetches live MR data, diff, and CI pipeline status via GitLab REST API
2 Chronos Matches the MR against historical incidents — "have we seen this pattern before?"
3 Athena Maps service topology, computes blast radius, detects breaking API changes
4 Cerberus Scans dependencies for CVEs, scans the diff for hardcoded secrets and auth regressions
5 Hermes Analyzes DB migrations, assesses deployment timing, evaluates rollback complexity
6 Morpheus Calculates 0–100 risk score and simulates the step-by-step production failure cascade
7 Echo Persists the analysis to Firestore organizational memory, surfaces team risk patterns
8 Prevention Generates runbook · creates GitLab issues · posts risk report · auto-blocks CRITICAL MRs · annotates risky diff lines inline · gates CI pipeline
9 Report Delivers structured FINAL_RISK_REPORT JSON with risk level, blocking issues, and recommendations

What a Real Analysis Looks Like

When MNEMOSYNE ran against a live production merge request during testing:

RISK SCORE: 86/100 — CRITICAL

⚠ MORPHEUS SIMULATION — MR #1 — 85% FAILURE PROBABILITY

T+0m   [payment-service]   Deployed. RS256 tokens sent to auth-service.
T+2m   [auth-service]      user_claim field rejected → 401 cascade begins.
T+3m   [payment-service]   Error rate 45%. All checkouts return HTTP 500.
T+4m   [api-gateway]       Circuit breaker opens. Users see 503.
T+9m   [ALL]               P1 declared. On-call SRE paged. $47K/min lost.
T+14m  [payment-service]   DB migration runs. Payments table LOCKED.
T+90m  [RESOLVED]          After 90-minute outage. $4.23M revenue lost.

⛔ MR BLOCKED — Converted to Draft. Merge button disabled.

5 GitLab issues were automatically created. An incident runbook was generated. A formatted risk report was posted directly to the MR — and the Merge button was disabled. All before anyone clicked Approve.

MNEMOSYNE Doesn't Just Analyze — It Acts

Most analysis tools surface findings and stop. MNEMOSYNE takes autonomous action through 7 prevention tools:

Tool Action
generate_incident_runbook Full on-call runbook — mitigation steps, rollback commands, escalation path, MTTR estimate — generated before the incident
create_gitlab_issues_for_findings Creates GitLab issues for every blocking finding: severity label, assigned to MR author, linked to analysis
post_risk_report_to_mr Posts structured markdown risk report to the MR — developers see score, blast radius, and blocking issues without leaving GitLab
block_high_risk_mr Converts any CRITICAL-risk MR (score ≥ 80) to Draft via /draft quick action — the Merge button is physically disabled
analyze_pipeline_failure Diagnoses failed CI jobs — fetches logs, identifies root cause, generates fix suggestions
post_inline_diff_comments Annotates specific risky lines directly in the diff using GitLab's discussions API — risk warnings appear on the exact changed line, eliminating context switching between the report and the code
mnemosyne-gate CI stage A .gitlab-ci.yml job that calls /api/analyze and fails the pipeline if risk score exceeds a configurable threshold — blocking merges at the CI level before anyone clicks Approve

Two Ways to Trigger

1. GitLab Webhook (automatic) — Fires on every MR open/update event. Zero developer action required.

2. GitLab Duo Agent Platform (on-demand) — Any team member can type in GitLab Duo Chat:

@GitLab Duo analyze this MR for risk: https://gitlab.com/org/repo/-/merge_requests/847

MNEMOSYNE runs the full analysis and returns the report — directly inside the GitLab UI where developers already work.

3. Pipeline Failure Diagnosis (reactive) — When CI breaks:

POST /api/pipeline-failure { "mr_url": "..." }

MNEMOSYNE fetches the failed job logs, diagnoses the root cause, and returns a fix — no manual log-reading required.

How We Built It

The AI Core — Google ADK + Gemini 2.5 Flash on Vertex AI

The intelligence layer is a single LlmAgent — MNEMOSYNE_Orchestrator — built with Google ADK 2.0.0 and powered by Gemini 2.5 Flash on Vertex AI. It holds all 29 tools and enforces the 9-phase analysis order through a structured system prompt that defines each specialist (Chronos, Athena, Cerberus, Hermes, Morpheus, Echo) as a named analysis role with specific tools and output formats.

A key architectural decision: Google ADK's transfer_to_agent() is a one-way handoff that terminates the runner. We tried multi-agent routing early on and hit this wall. The solution was to embed all six specialist concepts as named phases within one orchestrator — the agents are roles, not separate runner instances. This gave us full sequential control over the 9-phase pipeline without losing the agent reasoning depth.

ADK Runner → InMemorySessionService → MNEMOSYNE_Orchestrator
  → Vertex AI (gemini-2.5-flash, location: global)
  → 29 tools across 8 categories
  → 9-phase prompt enforcement
  → FINAL_RISK_REPORT JSON

The Backend — FastAPI on Cloud Run

The API layer is FastAPI with four surfaces:

  • POST /api/analyze — synchronous JSON endpoint (GitLab Duo Agent Platform)
  • WS /ws/analyze — WebSocket streaming endpoint (real-time dashboard)
  • POST /api/webhook/gitlab — GitLab webhook receiver (auto-trigger)
  • POST /api/pipeline-failure — reactive CI failure diagnosis endpoint

All deployed on Google Cloud Run (2 vCPU, 2Gi RAM, allow-unauthenticated) with auto-deploy wired through Cloud Build on every push to main.

Secret Manager holds the GitLab PAT. The Cloud Run service identity pulls it at runtime — zero credentials in the container image.

GitLab Integration — Duo Agent Platform + MCP Server + REST API

MNEMOSYNE integrates with GitLab at three levels, using the full breadth of GitLab's partner offering:

1. GitLab Duo Agent Platform — MNEMOSYNE is registered as a custom agent in GitLab's Duo ecosystem via .gitlab/duo/flows/mnemosyne.yaml. Any team member can invoke MNEMOSYNE directly from GitLab Duo Chat inside any MR — no external tools needed. The Duo Agent Platform uses MCP-compatible tool-calling protocols internally, making this MNEMOSYNE's primary GitLab MCP-based integration surface.

2. GitLab MCP Server — MNEMOSYNE connects to the hosted GitLab MCP server at gitlab.com/api/v4/mcp via ADK's MCPToolset with SseServerParams. When the Duo namespace is configured and GITLAB_MCP_ENABLED=true, ADK loads native GitLab MCP tools (MR data, diff, issues, pipeline status) and prepends them to MNEMOSYNE's 29 Python specialist tools.

3. 29 Specialist Python Tools — A purpose-built set of GitLab REST API tools covering the full GitLab surface: MR metadata, diff, CI/CD pipeline status, job log retrieval, commit history, branch comparison, issue creation, MR comment posting, inline diff annotations, and Draft conversion. These tools run every analysis phase and complement the MCP layer with capabilities specific to MNEMOSYNE's specialist analysis pipeline.

The Architecture

Nixora System Architecture

The Memory Layer — Google Cloud Firestore

MNEMOSYNE's Echo phase doesn't just log analyses — it persists them to Firestore as queryable organizational memory. Every analysis is stored with its risk score, detected patterns, and incident matches. The get_team_patterns tool queries Firestore to surface cross-MR trends: which categories are generating the most CRITICAL findings, which authors are repeatedly touching high-risk areas, whether auth regression rates are improving or worsening over time.

This is the institutional memory that Confluence pages promised but never delivered.

The Prevention Layer — 7 Autonomous Actions

Seven tools — none of which exist in any other MR analysis platform:

Tool Action
generate_incident_runbook Full on-call runbook with mitigation steps, rollback commands, escalation path, and MTTR estimate — generated before the incident
create_gitlab_issues_for_findings Creates GitLab issues for every blocking finding: severity label, assigned to MR author, linked to analysis
post_risk_report_to_mr Posts a structured markdown risk report comment to the MR — developers see risk score, blast radius, and blocking issues without leaving GitLab
block_high_risk_mr Posts /draft quick action to any CRITICAL-risk MR (score ≥ 80) — converts to Draft, disabling the Merge button
analyze_pipeline_failure Fetches failed CI job logs, diagnoses root cause with Gemini, generates fix suggestions
post_inline_diff_comments Annotates specific changed lines in the diff directly using GitLab's discussions API — risk warnings appear on the exact lines that triggered them, eliminating context switching
mnemosyne-gate CI stage A .gitlab-ci.yml job that calls MNEMOSYNE's analyze endpoint and fails the pipeline if risk ≥ threshold — CI-level merge blocking, not just a comment

Infrastructure

Component Technology
AI Engine Google ADK 2.0.0 + Gemini 2.5 Flash on Vertex AI
API FastAPI + uvicorn
Hosting Google Cloud Run (auto-scaling)
CI/CD GitLab CI → Cloud Build → Artifact Registry → Cloud Run
Secrets Google Secret Manager
Memory Google Cloud Firestore
Analytics Google BigQuery
Events Google Cloud Pub/Sub
Frontend Vanilla JS + D3.js
GitLab Duo Agent Platform + Webhook + REST API (httpx)

Challenges We Ran Into

1. ADK Multi-Agent Architecture Limitation

We originally designed MNEMOSYNE as six separate agents — each specialist as its own LlmAgent with its own runner. The moment we tested sequential handoffs, we hit the fundamental ADK constraint: transfer_to_agent() terminates the originating runner. There's no way to run six agents sequentially in one session with the current ADK design.

The fix required rethinking the architecture: collapse all six specialists into one orchestrator, encode their behavior as named phases in the system prompt, and use tool grouping to give each phase the right capabilities. The result was actually better — one runner means one coherent context window, and Gemini can see every previous phase's findings when running each new one.

2. Making Blocking Actually Block

The first version of block_high_risk_mr just posted a warning comment. Warning comments get ignored. The real blocker was finding a GitLab mechanism that would prevent the merge from happening without requiring repository admin access or branch protection rule changes.

The answer was the /draft quick action — a standard GitLab feature that converts a MR to Draft state, graying out and disabling the Merge button for all reviewers. Any user with Developer role can trigger it via a comment. This meant MNEMOSYNE could block merges with only the same permissions as a normal developer — no admin rights required.

3. Gemini 2.5 Flash Quota on First Live Runs

The very first time we switched DEMO_MODE=false and pointed MNEMOSYNE at a real MR, we hit 429 RESOURCE_EXHAUSTED. The 9-phase analysis is long — Gemini processes thousands of tokens across 27 tool calls. We implemented 3-attempt retry logic with 20s/40s exponential backoff in the /api/analyze endpoint and added intelligent fallback routing in the orchestrator.

4. Type System Collision — List[str] vs list

A subtle Python typing issue surfaced only when Gemini tried to call post_risk_report_to_mr with real data. The similar_incidents parameter was annotated as bare list, which Gemini's function-calling schema validation rejected with 400 INVALID_ARGUMENT: similar_incidents.items: missing field. Changing to List[str] fixed the schema — and exposed a second bug where the formatting code called .get() on strings that were now typed as str instead of dict. Both fixed with proper type guards.

5. GitLab MCP Transport Selection

Early design ran the GitLab MCP server as a local npx subprocess via StdioServerParameters. We hit two problems: the package was not published to the npm public registry, and the ADK MCPToolset.from_server() API changed between versions. The solution was to connect to GitLab's hosted MCP server at gitlab.com/api/v4/mcp using ADK's SseServerParams instead — no local subprocess, no npm dependency, cleaner architecture. The 29 specialist Python tools run alongside the MCP layer, covering analysis phases the GitLab MCP server doesn't expose (blast radius simulation, risk scoring, runbook generation).

6. Windows Development Quirks

Building a Linux Cloud Run container on Windows 10 introduced the usual friction: ngrok required an Administrator-level upgrade to reach version 3.20.0+, Git Bash curl commands need single-line syntax (no PowerShell backtick continuation), and the architecture diagram broke because Windows default fonts don't include emoji glyphs (fixed by replacing all emojis with 2-3 letter labels like "FS" for Firestore).

Accomplishments We're Proud Of

End-to-end live analysis works. We ran MNEMOSYNE against a real GitLab merge request with DEMO_MODE=false and watched all 9 phases execute against real data:

  • Risk score: 86/100 CRITICAL
  • 5 GitLab issues auto-created from the findings
  • MR comment posted automatically with the full risk report
  • Failure cascade predicted with step-by-step timeline and $4.23M revenue impact
  • Incident runbook generated with rollback commands before the merge
  • MR converted to Draft — Merge button disabled automatically

MNEMOSYNE blocks merges. Not warns. Not flags. Not suggests. For any MR scoring CRITICAL (≥ 80/100), it uses the GitLab /draft quick action to physically disable the Merge button — with no admin access required, no branch protection changes, just a well-placed API call.

MNEMOSYNE annotates the diff. Risk warnings appear on the exact changed lines that triggered them using GitLab's discussions API with diff position objects — no cross-referencing between a report and the code. The developer sees the risk on the code itself.

MNEMOSYNE gates the CI pipeline. The mnemosyne-gate stage in .gitlab-ci.yml calls /api/analyze on every MR pipeline and fails the build if risk exceeds threshold. Merge blocking at the pipeline level, visible in every CI run.

MNEMOSYNE diagnoses CI failures. The /api/pipeline-failure endpoint fetches failed job logs, identifies root cause, and generates fix suggestions — turning a 45-minute debugging session into a 30-second query.

Real Firestore organizational memory. The Echo phase now writes to and reads from Google Cloud Firestore. Analyses accumulate. Patterns emerge. Every future analysis benefits from every past one.

GitLab Duo Agent Platform integration works. MNEMOSYNE is registered as a custom agent, callable from GitLab Duo Chat on any MR in any project. The full 9-phase analysis runs and returns the risk report — inside GitLab, where developers already are.

Zero-credential container. The Cloud Run service identity pulls the GitLab token from Secret Manager at runtime. No secrets in the image, no secrets in environment variable plaintext visible in Cloud Console. Secret Manager handles rotation.

27 tools, one orchestrator, full sequential reasoning. The 9-phase prompt design means Gemini sees Chronos's incident match results when Morpheus runs the failure simulation. The historical context feeds the risk score. The risk score shapes the runbook priority. The whole pipeline is one coherent reasoning thread.

What We Learned

ADK is a powerful framework that rewards architectural clarity. The multi-agent approach felt natural to design — "six specialists" maps cleanly to how engineers think about incident analysis. But ADK's current transfer_to_agent() semantics forced us to find a better abstraction: named phases inside one orchestrator. The result is simpler, faster, and more coherent than our original design.

Gemini is an extraordinarily capable reasoning engine. Asking it to run a structured 9-phase analysis, call 27 tools in order, reason about historical incident patterns, compute blast radius, simulate failure cascades, diagnose CI job failures, and then write a usable on-call runbook — it does all of that without fine-tuning. The key is the system prompt architecture. Telling Gemini exactly which output format each phase must produce (tagged [CHRONOS], [ATHENA], etc.) makes the pipeline parse-friendly and the analysis disciplined.

Prevention > Detection. Every code review tool detects problems. The insight that changed our architecture was: detection without action puts cognitive load on the reviewer. MNEMOSYNE's value isn't the risk score — it's the runbook, the GitLab issues, the MR comment, and especially the Merge button that doesn't work. By the time a developer reads the risk report, the remediation work is already half-done and the risky merge is already blocked.

Blocking is a UX feature. Engineers don't ignore risk warnings because they're careless — they ignore them because warnings have no teeth. Converting an MR to Draft is the first mechanism in the pipeline that engineers physically cannot bypass without taking a deliberate action. That's the difference between a guardrail and a speed bump.

GitLab's API surface is exceptional. MR diffs, CI pipeline status, job log retrieval, issue creation, MR comments, Draft conversion — everything is well-documented and consistent. Building 27 tool wrappers took less time than expected because the REST API is predictably designed.

What's Next

Longitudinal memory — MNEMOSYNE currently stores analyses in Firestore and retrieves them. The next version uses BigQuery to run pattern analysis across all historical analyses: "Your team has a 34% higher auth regression rate than the org average. Here are the three MR patterns responsible."

GitLab CI/CD pipeline guard — Add a mnemosyne-gate CI stage that fails the pipeline if risk score exceeds a team-defined threshold. Blocking merges at the pipeline level, not just at the review level.

Fine-tuned Gemini model — Training on accumulated incident data and historical MR outcomes. MNEMOSYNE gets smarter every time a predicted failure matches a real incident (or doesn't).

Multi-provider support — The tool layer is GitLab-specific today. The orchestrator architecture ports cleanly to GitHub, Bitbucket, or Azure DevOps. The incident intelligence is platform-agnostic.

Slack/PagerDuty integration — Post risk reports to the team Slack channel. Auto-page the on-call if CRITICAL risk is detected on a merge to main during business hours.

Built With

  • d3.js
  • docker
  • fastapi
  • gemini-2.5-flash
  • gitlab
  • gitlab-duo-agent-platform
  • google-adk
  • google-artifact-registry
  • google-bigquery
  • google-cloud-firestore
  • google-cloud-pubsub
  • google-cloud-run
  • google-secret-manager
  • httpx
  • python
  • vertex-ai
  • websockets
Share this project:

Updates