OpsCopilot

Autonomous AI agent incident platform for SRE & DevOps powered by NVIDIA Nemotron on Nebius Token Factory & Tavily AI Search.

Comment

💡 Inspiration

When production outages strike at 3:00 AM, Site Reliability Engineers (SREs) and DevOps teams face an overwhelming barrage of raw, fragmented server logs, cryptic Kubernetes stack traces, and urgent Slack pings. Investigating the true root cause and formulating safe remediation scripts takes an average of 45–90 minutes of high-stress digging.

We built OpsCopilot to give on-call engineers an autonomous, supercharged AI copilot that ingests raw telemetry, conducts multi-hop causal reasoning using NVIDIA Nemotron models served on Nebius Token Factory, pulls live CVE and community issue intelligence with Tavily AI Search, and safely tests declarative fix scripts inside a protected virtual sandbox before anything touches production.

⚙️ What It Does

  1. Raw Log & Telemetry Ingestion: Ingests raw logs with 1-click incident presets (K8s OOMKilled, PostgreSQL Lock Contention, Redis Memory Saturation, Stripe TLS Timeouts, AWS IAM Denials) with automated PII and Bearer token scrubbing.
  2. Hermes 5-Phase Agent Orchestration:
    • Phase 1: Ingestion & Anomaly Signature Extraction: Isolates fatal markers, stack traces, and timestamp clusters.
    • Phase 2: Live Web Intelligence & CVE Check via Tavily AI Search: Queries live GitHub issues, CVE advisories, and official Kubernetes documentation to eliminate hallucinations.
    • Phase 3: Infrastructure Telemetry Correlation: Correlates container CPU/memory cgroups, socket states, and kernel OOM signals.
    • Phase 4: Causal Hypothesis Validation: Powered by NVIDIA Nemotron-3 Ultra on Nebius Token Factory for sub-2-second causal diagnosis.
    • Phase 5: Declarative Remediation Synthesis & Safety Checks: Generates Kubernetes manifests, shell scripts, and automated rollback configurations.
  3. Multi-Hop Escalation to Ultra Max: Deep causal re-analysis using NVIDIA Nemotron-4 340B Instruct for complex cascading failures.
  4. Isolated Virtual Sandbox Execution: Safely evaluates AI-synthesized fix scripts inside an isolated container sandbox protected by AST security policy blocklists (preventing destructive commands like rm -rf / or fork bombs).
  5. Multi-Platform Alert Cards: Instantly generates interactive Slack Block Kit and Microsoft Teams Adaptive Card webhook payloads with action triggers.
  6. SRE Telemetry Dashboard: Live monitoring of Mean Time to Resolution (MTTR), active vs resolved outages, and microservice health.
  7. 1-Click Postmortem Generator: Exports comprehensive, structured postmortem documentation in Markdown and JSON.

🛠️ How We Built It

  • AI Inference Engine: Powered by Nebius Token Factory (https://api.tokenfactory.nebius.com/v1) using nvidia/nemotron-3-ultra for rapid diagnosis and nvidia/nemotron-4-340b-instruct for complex multi-hop causal escalation.
  • Real-Time Grounding: Integrated Tavily AI Search API for live web intelligence against official Kubernetes docs, CVE databases, and GitHub issues.
  • Backend Architecture: Node.js & Express with better-sqlite3 in WAL mode, JWT authentication, and token-bucket rate limiters.
  • Frontend UI/UX: React 19, TypeScript, Tailwind CSS, Vite, and Lucide Icons with dark-mode glassmorphism.
  • Host Security: Zero frontend AI provider keys (all requests proxied via backend), automatic PII sanitization, and AST command execution blocklists.

🏔️ Challenges We Ran Into

  • Log Noise & Context Window Efficiency: Raw production logs often contain thousands of repetitive INFO lines. We engineered an intelligent log sanitizer and signature extractor that isolates critical stack traces and timestamp clusters before feeding context into Nemotron.
  • Agent Output Reliability: SRE workflows require exact JSON schemas. We crafted structured prompt templates with strict type definitions to guarantee parseable JSON responses containing scripts, confidence scores, and reasoning steps.
  • Execution Safety: Giving an AI agent the ability to generate bash scripts poses security risks. We enforced strict AST blocklists and isolated container sandboxes so no AI code executes directly on the host machine.

🏆 Accomplishments We're Proud Of

  • Sub-2-Second Incident Diagnosis: Combining Nebius Token Factory's fast TTFT with Nemotron-3 Ultra delivers full causal root-cause diagnoses in seconds.
  • 10/10 Automated Integration Tests: Built an automated integration test suite verifying healthchecks, auth, AI diagnosis, Tavily search, sandbox isolation, security guards, and escalation.
  • Complete End-to-End Product Experience: Delivered a polished, fully functioning product with interactive sandbox runners, live alerts, and telemetry analytics.

📚 What We Learned

  • How to design robust multi-hop agent reasoning architectures using the Hermes pattern.
  • The efficiency and reasoning speed advantages of NVIDIA's specialized Nemotron models on Nebius high-performance infrastructure.
  • The value of real-time search grounding with Tavily to eliminate hallucinations during complex infrastructure troubleshooting.

🔮 What's Next for OpsCopilot

  • Direct integration with OpenTelemetry and Datadog / Prometheus webhooks for automated zero-touch incident ingestion.
  • Multi-cluster Kubernetes operator deployment for automatic canary rollouts of AI-synthesized patches.
  • Interactive Slack Bot and Microsoft Teams bot for 1-click approvals right from chat channels.

Built With

Share this project:

Updates

Submission history