Unified Ops AX

• 

  streamlit app

• 

  CloudRun Observability

• 

  CloudRun map

• 

  CloudRun console

• 

  Yaml

• 

  Dashboard

🚀 Unified Ops AX: Autonomous Fleet Telemetry & Governed Multi-Agent Engine

🎯 Inspiration

Modern enterprise AI aplications generate millions of API calls, log events, and telemetry data points every day. However, managing AI agent fleets at scale presents severe operational bottlenecks: unexpected cost spikes from large LLM models, latency bursts during peak traffic, security DLP/PII violations, and manual incident response delays.

When an LLM model experiences an error burst or cost runaway, human operators cannot react fast enough. Furthermore, in enterprise environments, unconstrained AI agents pose severe compliance risks if they perform unauthorized mutations or bypass security boundaries.

We built Unified Ops AX for the All Things Agentic Hackathon to solve this core challenge:

Can we build autonomous background agents that heavy-lift massive telemetry log streams, enforce hard security boundaries in code rather than prompts, and execute self-healing remediation policies asynchronously without blocking human operators?

---

☁️ Mandatory Google Cloud Infrastructure & AI Tech Proof

Per official hackathon rules ("Mandatory for all categories: at least one Google Cloud infrastructure service such as Cloud Run, Cloud SQL, Firestore, GKE, Pub/Sub"), Unified Ops AX natively leverages 6 Google Cloud Infrastructure & AI services:

1. Google Cloud Run (Serverless Container Host)
   • Live Service URL: https://unified-ops-ax-652787573242.us-central1.run.app/
   • Role: Auto-scaling serverless HTTP container hosting FastAPI, Uvicorn, 5 Governed AI Agents, and MCP endpoints.
2. Vertex AI & Gemini Models
   • Models: Pinned gemini-3.5-flash via Vertex AI Generative AI SDK for agent reasoning and text-embedding-004 for production RAG vector search.
3. Google Cloud Pub/Sub (Activity Event Bus)
   • Topic: projects/agentichackathon-506620/topics/activity-events
   • Role: Asynchronous transactional outbox publisher for real-time enterprise event streams (as.opened, as.resolved, delivery.done, order.placed, evolve.audit).
4. Google Cloud Firestore (NoSQL Document Store & Audit Chain)
   • Collection: activity_logs
   • Role: Persistent state storage and tamper-evident audit logging for all agent actions.
5. Google Cloud Storage (GCS RAG Asset Bucket)
   • Bucket: gs://agentichackathon-506620-rag-docs
   • Role: Document lake storage for enterprise knowledge bases indexed into vector search.
6. Google Cloud SQL (PostgreSQL with pgvector)
   • Instance: agentichackathon-506620:us-central1:unified-ops-db
   • Role: Relational Single Source of Truth (Activity store) with SQL-level Row-Level Security (RLS) & security trimming before vector top-k retrieval.

---

🛡️ What It Does

Unified Ops AX is an AI-Powered Autonomous Fleet Telemetry & Governed Remediation Engine. It acts as an intelligent, self-healing operational desk powered by 5 Governed AI Agents:

1. AS Triage Agent (as.opened): Classifies incoming support tickets, determines severity, and auto-assigns to team members based on workload.
2. Knowledge Agent (as.resolved): Converts resolved support tickets into structured knowledge documents and indexes them into vector RAG.
3. Follow-up Agent (delivery.done): Drafts customer follow-up messages — enforced by a strict Human-in-the-Loop (HITL) gate (cannot send directly; unapproved sends fail deterministically with HTTP 409 Conflict).
4. Reconcile Agent (order.placed): Checks the order book against accounting ledgers in a zero-side-effect, read-only mode.
5. Evolve Agent (evolve.audit): Systematically probes application links, endpoint latencies, PII encryption status, and issues automated self-healing improvement directives.

Key Engine Features:

• Autonomous Asynchronous Engine (AsyncAgentEngine): Priority queue workers (CRITICAL, HIGH, NORMAL, LOW) executing background jobs non-stop.
• Real-Time Telemetry & Anomaly Alerts: Captures token consumption, latency, cost USD, router decisions, and DLP PII security rule violations.
• Event-Driven Auto-Remediation (auto_remediation.py): Reacts instantly to anomaly alerts by dynamically tuning model weights, opening circuit breakers, and executing fallback routes.

---

🛠️ How We Built It

We architected Unified Ops AX using a 5-Layer Modular Monolith architecture:

• AI Foundation: Built on Google Cloud Vertex AI with Gemini 3.5 Flash and Google Agent Developer Kit (ADK) agent-to-agent capabilities.
• Core Engine: Built with Python 3.11, FastAPI, and asyncio.PriorityQueue to manage asynchronous worker threads and non-blocking job execution.
• Security Trimming: Authorization is enforced in SQL (WHERE clause security trimming) before vector similarity ranking, preventing privilege escalation regardless of prompt input.
• Model Context Protocol (MCP): Native MCP server (app/mcp/) exposing 7 read-only internal tools over JSON-RPC / stdio for external AI client integration.
• Dashboards: Built both a Streamlit Control Desk (https://unified-ops.streamlit.app/) and a FastAPI Web UI.

---

💡 Challenges We Ran Into

1. Asynchronous Queue Concurrency under High Load: Ensuring high-priority anomaly remediations (CRITICAL) jump ahead of routine telemetry indexing (NORMAL) without starving worker tasks required fine-tuning our priority queue state machine.
2. Strict Security Boundaries in Code: Preventing LLMs from forging user identity required deriving roles on the server side — no tool accepts a role or identity parameter from the model.
3. Dual Database Driver Resiliency: ADK session storage uses asyncpg while domain logic uses pg8000. We unified socket configuration to allow seamless offline SQLite fallback for testing alongside production Cloud SQL Postgres.

---

🏆 Accomplishments That We're Proud Of

• ⚡ High Throughput Performance: Achieved > 31 tasks/sec queue throughput, indexing 1,000 telemetry events (512 KB) in 0.317 seconds.
• ⏱️ Sub-10ms Auto-Remediation: Instant background policy execution when alerts trigger, switching fallback routes and restoring baseline weights automatically.
• 🧪 100% Verification Rate: Comprehensive automated test suite with 103 unit tests passed and 17/17 E2E verification checks passed.
• 🎥 Authentic Google Cloud Proof Video: Generated a 72-second narrated video showcasing live Google Cloud Console Cloud Run service revisions (unified-ops-ax-00015-8pd), source code tree, and real-time metrics. https://youtu.be/5W6H2g0u0PM

---

📚 What We Learned

• "Code is the Contract; Instruction is a Courtesy": Safety and security boundaries must be enforced by Python/SQL logic rather than relying on prompt instructions.
• Background Agents are Essential for Fleet Scale: Offloading heavy lifting to asynchronous background workers drastically improves system responsiveness.
• Decoupled Tool Contracts: Structuring agent tools with explicit schemas ensures seamless execution across local offline fallbacks and remote multi-LLM endpoints.

---

🌐 What's Next for Unified Ops AX

• Redis Outbox Draining & Vector Caching: Implementing Redis Pub/Sub outbox draining for sub-10ms RAG retrieval under high concurrency.
• WebSocket 3D Map Streaming: Adding FastAPI WebSocket streams for real-time 3D spatial unit map updates.
• Automated GCP KMS Key Rotation: Integrating GCP Cloud KMS for automated 90-day AES-GCM PII key rotation.

---

⚡ Live Project Links

• Streamlit Control Desk: https://unified-ops.streamlit.app/
• Google Cloud Run Backend: https://unified-ops-ax-652787573242.us-central1.run.app/
• FastAPI Interactive Docs: https://unified-ops-ax-652787573242.us-central1.run.app/docs
• Cloud Run Observability Dashboard: Google Cloud Console — Cloud Run Metrics
• GitHub Repository: https://github.com/sechan9999/unified-ops-ax

-Youtube: https://youtu.be/1UAzXaKZwq8?si=d9L7a6WsdhM_MnFv

Built With

• cloud-run
• docker
• eventarc
• fastapi
• gemini
• google-adk
• google-cloud
• kubernetes
• prometheus
• pubsub
• pydeck
• pytest
• python
• streamlit
• vertex-ai