AgentShield AI

AI incident remediation platform powered by NVIDIA Nemotron on Nebius Token Factory. Combines verifiable log evidence, sandboxed patch validation, and human sign-off.

Comment

Inspiration

When production outages strike at 3:00 AM, engineering teams face high cognitive load, fragmented observability dashboards, and cascading error logs. While generic LLMs have been pitched as on-call assistants, they suffer from a fatal flaw: hallucination. Standard language models routinely invent plausible-sounding log lines, fabricate stack traces, or cite nonexistent code files. In mission-critical infrastructure, an unverified guess can turn a minor degradation into a catastrophic outage.

We built AgentShield to solve this: an evidence-driven AI incident investigator and remediation platform. Our core philosophy is simple: Zero fabricated evidence. Every hypothesis must be grounded in authentic ingested logs, verified against real line numbers, and gated behind isolated sandbox tests and explicit human sign-off before any code touches staging or production.

What It Does

AgentShield automates the incident diagnosis lifecycle through an orchestrated 7-stage pipeline:

  1. Deterministic Log Ingestion & Anomaly Detection: Parses unstructured log streams, normalizes timestamps, clusters error templates by fingerprint, and computes baseline spike multipliers \( S = \frac{E_{\text{peak}}}{E_{\text{baseline}}} \).
  2. Context Retrieval: Filters priority error traces, exception frames, and correlated request IDs across affected services.
  3. Runbook & Knowledge Correlation: Cross-references internal architectural runbooks and postmortems.
  4. NVIDIA Nemotron Reasoning: Leverages NVIDIA open-source reasoning models (served on Nebius Token Factory) to generate candidate root-cause hypotheses.
  5. Deterministic Evidence Verification: Every citation is verified against real file line numbers (e.g., payment-gateway.log:5). The system calculates a deterministic confidence score \( C_{\text{det}} \) distinct from subjective model confidence: $$C_{\text{det}} = \text{clamp}\left(50 + 15 \cdot |E_{\text{sup}}| - 25 \cdot |E_{\text{con}}|, \; 5, \; 98\right)$$ where \( |E_{\text{sup}}| \) is the number of verified supporting citations and \( |E_{\text{con}}| \) is the number of contradicting evidence items.
  6. Code Patch Laboratory: Generates minimal safe git diffs, verifies AST syntax and timeout constraints, and executes unit tests in an isolated sandbox runner without network egress.
  7. Human-in-the-Loop Approval & Postmortem: Mandates explicit Approve, Reject, or Request Changes authorizations, updating a tamper-evident audit ledger and generating downloadable Markdown postmortems.

How We Built It

  • AI Inference & Models: Powered by NVIDIA open-source models (NVIDIA Nemotron-70B-Instruct) hosted on Nebius Token Factory (https://api.studio.nebius.ai/v1). We built a dual-provider abstraction with:
    • A real Nebius Provider using official OpenAI-compatible endpoints with Bearer authentication, latency tracking, and token usage accounting.
    • A Deterministic Mock Provider for zero-dependency local testing, reproducibility, and CI benchmarks.
  • Frontend: React 19, TypeScript, Tailwind CSS, and Lucide React designed with a dark navy developer aesthetic, cyan accents, compact data tables, and readable unified diff viewers.
  • Backend Architecture:
    • Full-stack Node.js / Express runner with Vite middleware integration on port 3000.
    • Python FastAPI backend package (/backend) with SQLite3 repository, Pydantic v2 validation models, and automated test suites (backend/test_agent.py).
  • Benchmarking & Evaluation: An integrated 10-scenario synthetic evaluation suite testing failure modes (connection pool exhaustion, V8 heap memory leaks, TLS cert expiry, 429 rate limits, clock skew, deadlock cycles, and replication lag) with automated category accuracy and 100% citation validity checks.

Challenges We Faced

  1. Eliminating LLM Citation Hallucinations: Large models frequently cite generic lines (e.g. line 42) even when prompted to cite only provided logs. We engineered a deterministic post-inference cross-validation engine that maps each citation against the physical log array. If an LLM cites a line number or string not present in the ingested payload, it is rejected.
  2. Hypothesis Revision on Contradictory Evidence: Real incidents often present red herrings (e.g. initial cache latency warnings). We constructed demo scenarios where diagnostic probes (such as REDIS INFO hit ratio 99.6%) refute the initial hypothesis, requiring the agent to demote the refuted cause and promote the true root cause (read-replica replication lag).
  3. Safe Sandboxed Execution: Executing untrusted AI-generated code required strict boundaries. We implemented static AST guards to block unauthorized child processes (eval(), child_process) and isolated VM execution with execution timeouts.

What We Learned

  • Verification Beats Raw Generation: In DevOps and reliability engineering, an AI model that produces 3 verified citations with explicit uncertainty is vastly more valuable than a model that writes 50 lines of unverified speculation.
  • NVIDIA Nemotron Reasoning on Nebius: Deploying NVIDIA open-source models via Nebius Token Factory provided rapid inference latency (<600ms) with adherence to structured JSON schemas.
  • Human Agency as First-Class Architecture: Autonomy should stop at recommendations. Real engineering teams demand explicit sign-off controls before any configuration or deployment changes occur.

What's Next for AgentShield

  • Live OpenTelemetry & Prometheus Ingestion: Native connectors for Datadog, Grafana Loki, and AWS CloudWatch.
  • Automated Staging Canary Replay: Replaying sanitised traffic through patch verification containers.
  • Multi-Agent Consensus: Combining multiple specialized NVIDIA models (e.g., security auditor agent + database optimizer agent) to cross-review remediation patches.

Built With

Share this project:

Updates

Submission history