FlowGuideAI Ops Fleet

• 

  The end result of the question "What happened to checkout?" (it went down and here's the problem found - will add JIRA component later)

Inspiration

Every ops team has the same debt. Something breaks at 2 a.m., someone fixes it, and the incident report, the runbook update and the change record never get written. Not because anyone is lazy — because at 2 a.m. the fix is the job and the paperwork is tomorrow's problem, and tomorrow there's another fire.

Then the SOC 2 auditor asks for the change record and nobody has one.

The unlikely hero here is the on-call sysadmin and the MSP tech. They already did the hard part. What they need isn't an assistant that helps them write documents faster — it's a fleet that writes the documents without them, from the evidence they already produced: the production log entry itself.

## What it does

Ops Fleet turns one production error into verified, control-tagged, approval-routed
documentation with zero human turns.

A real Cloud Run service throws a real error. A Cloud Logging sink pushes the entry
to Pub/Sub, which wakes a fleet of six agents:

  incident_watcher   classifies the event and extracts only what it actually states
  drafter            writes the documents into FlowGuideAI via MCP
  verifier           reads back what was saved and rejects anything it can't verify
  compliance_mapper  tags the documents with the controls they evidence
  approval_router    routes them into the right human approval chain
  fleet_orchestrator answers the engineer's questions and holds durable org memory

The interesting one is the verifier. The drafter is a language model, so it will
occasionally invent a plausible hostname. The verifier reads the saved document back
out of FlowGuideAI and rejects it if any host, command, port or file path appears
that isn't in the triggering event or in organization memory. The drafter fixes
exactly those issues and re-saves. Only a verified document reaches an approval
queue.

That reject-and-redraft loop is the whole product. An ops document that confidently
cites a server which does not exist is worse than no document at all — it will be
followed at 3 a.m. by someone who trusts it.

How we built it

Google ADK 2.7.1 on Cloud Run, Gemini 3.7 Flash for the reasoning roles and 3.5 Flash-Lite for classification and routing, Firestore for the agent registry, incidents and memory, Pub/Sub for the async runtime with a dead-letter queue, a Cloud Logging sink as the trigger, and Secret Manager for credentials.

Guardrails sit at two boundaries — the engineer's typed prompt and, more importantly, the raw production log payload before any model reads it. Both call sites run Google Cloud Model Armor, screening through a template with the prompt- injection/jailbreak and malicious-URI filters enabled. Verified end to end: the injected log line returns MATCH_FOUND and the incident is recorded as blocked and acknowledged rather than retried; a genuine payment-gateway error returns NO_MATCH_FOUND and flows through to documentation.

The fleet talks to FlowGuideAI through its existing MCP server over Streamable HTTP, which is what gives it real document storage, real compliance mapping and real approval chains rather than a simulation of them.

Separation of duties is enforced by wiring, not by prompt text. Each agent is handed an MCP toolset whose tool_filter is derived from its entry in the Firestore agent registry. The verifier physically cannot call fgai_save_ai_draft — the tool is not in its toolset. "The verifier never edits" is a property of the plumbing, not a hope about the model's behaviour.

The registry is fail-closed: if it can't be read, every tool call is denied, and the worker refuses to start if any agent in its pipeline is unregistered or unapproved.

## Challenges we ran into

Guardrails at the boundary that actually matters. Screening the human's prompt is
table stakes. The interesting boundary is the production log payload — anything
that can write to your logs can attempt prompt injection against a fleet that reads
them. Ops Fleet screens the raw log entry through Model Armor at that boundary
before any model sees it, and a blocked event is acknowledged rather than retried,
because retrying an
attack just runs the attack again.

Making the fleet testable without burning the demo. Nearly everything was proven
before a single API call: a scripted model implementing ADK's BaseLlm drives the
real pipeline, the real MCP toolset and a real MCP server end to end with no key.
That harness caught three genuine bugs — a LoopAgent iteration bug where the drafter
skipped its redraft, an ASGI middleware bug that would have broken every real
orchestrator turn in production, and a plugin that was silently failing to load.
All three would otherwise have surfaced during the live demo.

ADK has sharp edges worth knowing. Agents are single-use — the framework stamps a
parent on composition, so building a pipeline twice raises a validation error;
they have to come from factories. Per-agent tool responses accumulate across loop
iterations, so a naive step counter breaks on the second pass. And the Cloud Trace
exporter opens a gRPC channel at import time, so it fails on a laptop unless it's
gated.

Accomplishments that we're proud of

The fleet refuses to lie. The verifier reads each saved document back out of storage and judges it against the organization's confirmed inventory — not by asking the drafter to grade its own homework. In practice the drafter holds the line: it marks infrastructure it cannot confirm as UNCONFIRMED rather than stating it as fact, so live runs approve on the first pass. When a drafter does slip, the verifier rejects it quoting the offending line and the redraft is checked again — demonstrated by deliberately injecting a faulty drafter and pointing the real verifier at it (3/3 runs).

Least privilege is real, not decorative. One service account per runtime. The orchestrator's identity cannot request approvals; the worker's cannot publish to Pub/Sub. The worker isn't publicly reachable at all — only Pub/Sub push can invoke it.

And the whole thing runs with zero human turns. A log line becomes three verified, control-tagged documents sitting in an approval queue before anyone wakes up.

## What we learned

Verify the platform, don't trust the docs. Every API assumption in this project was
confirmed by running it. That caught a dependency pin where installing the obvious
package breaks the build, a plugin that isn't exported where its documentation
implies, and a templating engine whose behaviour with literal JSON braces had to be
checked rather than assumed.

And separation of duties belongs in the wiring. Every time we were tempted to
express a constraint as an instruction — "you must not edit documents" — it was
stronger as a tool filter. Prompts are advisory. Toolsets are not.

What's next for FlowGuideAI Ops Fleet

Post-incident review on a timer — a follow-up agent that revisits an incident seven days later and asks whether the runbook it wrote actually held up.

Scope enforcement end to end, so an integration approved for read access is mechanically prevented from writing.

And widening the trigger surface beyond Cloud Logging: alerting webhooks, CI failures, and change events from the deployment pipeline.

Required disclosure

FlowGuideAI and its MCP server are pre-existing products of All Aligned Consulting LLC and were not built for this hackathon. Everything in the fgai-ops-fleet repository was: the six-agent fleet, the Firestore agent registry and gateway, the Model Armor guardrails, the Pub/Sub runtime, the per-agent service-account model, and the demo service. The fleet consumes FlowGuideAI purely as an external MCP tool provider, over the same public interface any third party would use.

Category justification — Fortified Enterprise Fleet

Agent Registry Firestore agent_registry/{name}: version, owning service account, allowed tools, approval status. The worker refuses to start if any agent in its pipeline is missing or unapproved. → ops_fleet/registry.py, infra/registry.json

Agent Gateway gateway_check(agent, tool) before dispatch, plus a tool_filter on every MCP toolset derived from that agent's registry entry. Fail-closed: an unreadable registry denies every call. → ops_fleet/registry.py, ops_fleet/fgai_tools.py

Agent Identity One service account per runtime, least privilege. The orchestrator SA cannot request approvals; the worker SA cannot publish to Pub/Sub. The worker is --no-allow-unauthenticated and is invocable only by the Pub/Sub push identity. → infra/setup.sh, infra/deploy.sh

Agent Runtime Async and long-running by construction: Cloud Logging sink → Pub/Sub → push handler. Incidents persist in Firestore; 900s timeout; three delivery attempts then a dead-letter queue. → worker/main.py, infra/deploy.sh

Memory Bank Firestore memory/{org_id}: durable environment facts and prior decisions, injected into every agent — and used by the verifier as a source of truth, which is what makes "no invented hosts" enforceable rather than aspirational. → ops_fleet/memory.py

Guardrails Model Armor at two boundaries: the engineer's typed prompt, and the raw production log payload before any model reads it. → ops_fleet/model_armor.py

Observability AutoTracingPlugin + trace_to_cloud → Cloud Trace; structured logs keyed by incident_id. → worker/main.py, orchestrator/main.py

Failure-tolerant Verifier can reject the drafter (max 2 iterations); poisoned events routing
are acked not retried; genuine failures retry then DLQ. → worker/pipeline.py, worker/main.py

Built With

• firestore
• gemini3.7