argus-k8s

Always watching. Never sleeping.

Comment

Inspiration

The idea started from my experience at IBM, where I worked with a synthetic testing framework that simulated customer behavior and helped uncover failures before customers encountered them.

The testing was valuable, but responding to a failure was still painfully manual. I saw how much time went into collecting logs, passing information between teams, comparing symptoms across services, and trying to reconstruct what had actually gone wrong.

Finding a bug was only the beginning. The harder part was identifying the real root cause, figuring out which dependency failed first, understanding the blast radius, and deciding how to recover. That process could take far too long.

It made me ask:

What if the system could do more than generate a failure and collect logs? What if it could investigate the evidence, reconstruct the root cause, understand the blast radius, recover safely, and verify the result automatically?

Around the same time, I became increasingly interested in Kubernetes security. I kept thinking about how many security products wait until suspicious behavior reaches the application or generates a conventional alert. By then, it may have already crossed several boundaries.

Kubernetes gives me several opportunities to intervene earlier:

  • Reject an unsafe workload at admission.
  • Detect suspicious behavior at the kernel or container runtime.
  • Restrict malicious communication at the network layer.
  • Isolate the affected workload before the threat spreads.

That idea became Argus, a layered Kubernetes security agent built around one principle:

Detect the problem at the earliest layer that can see it, contain it there, and preserve the evidence needed to explain the decision.

Once I had applied that principle to security, I realized I could apply the same idea to reliability.

Instead of waiting for a customer to discover a failure, Phoenix behaves like the customer first. It runs synthetic customer journeys and bounded chaos experiments against the system. When something breaks, Phoenix does not stop after reporting an error. It analyzes the evidence, reconstructs the root cause, predicts the blast radius, chooses a risk-aware recovery, and verifies that the service is healthy again.

Argus and Phoenix approach the same problem from two directions:

  • Argus asks: Can I stop a threat before it spreads?
  • Phoenix asks: Can I discover and repair a weakness before a customer feels it?

I built Sentinel to connect them through the Sentinel Operations Graph, creating one evidence and governance layer for security, resilience, root-cause analysis, recovery, and human approval.

The result is the platform I wished I had during those long debugging cycles: something that does not simply collect more logs or generate another alert, but turns evidence into a decision and follows that decision all the way through verified recovery.

What it does

Argus-k8s has grown into a three-agent security and resilience platform:

  • Argus detects and contains Kubernetes security threats.
  • Phoenix discovers failures, performs root-cause analysis, and verifies recovery.
  • Sentinel connects their evidence and coordinates operational decisions.

Together, they form one protection loop across admission, kernel/runtime, network enforcement, and service resilience.

Argus: layered threat detection and containment

Argus protects Kubernetes across several operational layers:

  • Kyverno evaluates workloads at admission and prevents unsafe configurations from entering the cluster.
  • eBPF and Falco observe kernel and container-runtime behavior, including suspicious processes, shell execution, sensitive file access, and other runtime activity.
  • Cilium provides network visibility and enforcement across L3, L4, and L7.
  • The Argus agent enriches detections, evaluates severity and blast radius, recommends a response, and preserves the evidence behind that recommendation.

The important difference is where Argus can respond. It does not always have to wait for an application-level alert. It can identify and contain suspicious behavior at the earliest capable enforcement layer.

Phoenix: synthetic chaos, error analysis, and root-cause resolution

Phoenix continuously tests whether services can survive realistic failure conditions.

For the portable demo, it can run deterministic simulations without requiring Kubernetes. In the live environment, it can create explicitly authorized Chaos Mesh experiments inside an isolated namespace.

These experiments cover bounded conditions such as:

  • Pod failure
  • Network latency
  • Packet loss
  • Infrastructure-operation failures
  • Dependency degradation

After detecting a fault, Phoenix brings together:

  • Metrics and logs
  • The active fault and affected resource
  • Service dependencies
  • The predicted blast radius
  • Previous incident outcomes
  • Current infrastructure state
  • Available remediation actions

OpenAI reasoning converts this evidence into a structured diagnosis containing:

  • The causal chain
  • The likely root cause
  • The affected services
  • The predicted blast radius
  • A recommended recovery action
  • The risk of executing that action
  • The reasoning behind the recommendation

Phoenix then follows a controlled lifecycle:

Detect → Diagnose → Plan → Approve when required → Execute → Verify → Report

I did not want Phoenix to call something “recovered” just because it dispatched an action. It performs a final health verification and records the result. If the action is high-risk or could affect multiple services, it waits for explicit human approval.

Sentinel: one operational command center

The Sentinel Operations Graph, or SOG, is the shared evidence layer that connects Argus and Phoenix.

It maps:

  • Services
  • Namespaces
  • Dependencies
  • Security findings
  • Resilience findings
  • Incidents
  • Recovery outcomes
  • Trust state

Sentinel uses the SOG to:

  • Correlate Argus threats with Phoenix resilience outcomes
  • Connect findings to affected services and dependencies
  • Rank resources by current risk
  • Produce evidence-backed operator briefings
  • Explain why a risk score or recommendation was generated
  • Route high-risk actions through human approval
  • Display the complete incident lifecycle
  • Preserve one audit trail across detection, diagnosis, recovery, and verification

I also wanted the platform to be honest about its evidence. Every record carries a visible provenance label:

  • Live Observed
  • Live Chaos Mesh
  • Synthetic Simulator
  • Replayed Evidence

This prevents synthetic demo evidence from being mistaken for a real production attack or outage.

The platform supports two demonstration modes:

  1. Portable judge demo: Runs the real local Argus, Phoenix, Sentinel, and SOG services using clearly labeled replay and simulation. It does not require Kubernetes.
  2. Guarded live k3s demo: Uses a real three-node k3s cluster with Kyverno, Falco, Cilium, and Chaos Mesh. It requires explicit authorization, isolates changes inside a dedicated namespace, measures recovery, and cleans up only the resources it created.

How I built it

The platform is split across four connected repositories:

  • argus-k8s for layered Kubernetes security
  • phoenix for chaos testing, error analysis, RCA, and verified healing
  • sentinel for orchestration and the unified command center
  • sentinel-platform for shared event contracts, adapters, and the SOG

I built the backend services primarily with Python and FastAPI. The dashboards use React, TypeScript, and Vite. Redis provides shared operational state.

The live Kubernetes environment integrates:

  • k3s
  • Kyverno
  • Falco
  • eBPF
  • Cilium
  • Prometheus
  • Loki
  • Chaos Mesh

How I used OpenAI

OpenAI reasoning operates inside two evidence-bounded workflows.

Phoenix uses OpenAI for root-cause analysis. It supplies the model with structured scenario evidence, dependency context, blast-radius information, operational history, metrics, and logs. The result is a causal chain, risk classification, action target, remediation recommendation, and explanation.

Sentinel uses OpenAI for cross-agent reasoning. It turns current SOG evidence into incident assessments, fleet-risk explanations, threat-hunting assistance, and operator briefings.

I deliberately kept the model separate from unrestricted execution. Deterministic policy governs what can happen. Malformed model responses fail closed, and high-impact actions require explicit human approval.

How Codex accelerated the project

I used Codex as an engineering partner throughout OpenAI Build Week.

Codex helped me:

  • Analyze four connected repositories
  • Decide how the three agents should communicate
  • Design shared evidence and provenance contracts
  • Implement the Sentinel Operations Graph integration
  • Build guarded k3s and Chaos Mesh workflows
  • Add Phoenix persistence across process restarts
  • Develop deterministic and randomized threat simulations
  • Diagnose backend, frontend, API, and cluster failures
  • Improve the Argus, Phoenix, and Sentinel dashboards
  • Build operator-ready incident reports
  • Add the seven-stage resilience proof timeline
  • Create presentation preflight checks
  • Write automated tests
  • Consolidate the setup documentation
  • Create the judge demo, evidence scorecards, and submission materials

The project existed before Build Week as separate security and resilience components. During Build Week, I used Codex to meaningfully extend it into a cohesive platform with cross-agent correlation, provenance-aware incidents, verified recovery, persistent history, human governance, one-command demonstrations, and judge-ready proof artifacts.

I tracked each phase through:

Issue → Branch → Implementation → Test → Commit → Pull Request

I made the product, safety, and architecture decisions. Codex helped me investigate problems, implement those decisions, test the results, and iterate much faster than I could have alone.

Challenges I ran into

Making autonomy trustworthy

The hardest part was not executing an action. It was deciding when the platform should be trusted to act.

Before taking action, the system needs to understand:

  • What evidence was actually observed
  • What evidence came from a simulator
  • Which service was the likely root cause
  • Which downstream services might be affected
  • Whether the proposed recovery was bounded
  • Whether a person needed to approve it
  • Whether the recovery actually worked

Solving this required explicit provenance, risk-aware routing, human approval boundaries, and verification after execution.

Connecting different evidence models

Argus produces security findings. Phoenix produces resilience scenarios, diagnoses, actions, and recovery outcomes. Kubernetes, Falco, Cilium, Prometheus, and Chaos Mesh all expose different kinds of data.

The SOG had to connect those records to the same services, namespaces, dependencies, incidents, and trust state without losing their original meaning.

Preventing synthetic-data ambiguity

The portable demo is useful because a judge can run it without building a cluster. However, I never wanted synthetic evidence to look like a real attack or production outage.

Every record therefore carries visible provenance. The portable scorecard reports availability as not measured instead of turning a simulator result into a false availability claim.

Building safe live automation

The live k3s demonstration creates real workloads and Chaos Mesh resources. That meant I needed strict safety boundaries:

  • Exact Kubernetes-context confirmation
  • An isolated demo namespace
  • Explicit authorization phrases
  • Security-stack preflight checks
  • Bounded experiments
  • Continuous health probes
  • Cleanup restricted to resources created by the demo

Making the demo reliable

A multi-service demo can fail for many boring reasons: occupied ports, missing dependencies, stale processes, unavailable APIs, cluster configuration, or frontend/backend mismatches.

I added one-command startup, non-mutating doctor checks, presentation preflight, deterministic seeded evidence, persistent history, dynamic demo feeds, and judge-readable proof artifacts to make the experience repeatable.

Making complex infrastructure understandable

The platform contains a lot of operational data, but that data is not useful if someone cannot understand the dashboard.

I went through several UI iterations to improve typography, spacing, navigation, provenance labels, incident reports, risk explanations, lifecycle visualization, and the relationship between Argus, Phoenix, and Sentinel.

Accomplishments I’m proud of

  • Built layered Kubernetes protection across admission, kernel/runtime, and network enforcement
  • Connected Kyverno, eBPF, Falco, and Cilium through one Argus evidence pipeline
  • Built Phoenix synthetic customer simulation and bounded chaos testing
  • Added OpenAI-powered Phoenix error analysis and root-cause reconstruction
  • Added dependency-aware blast-radius reasoning
  • Implemented risk-scored recovery recommendations
  • Added human approval for consequential actions
  • Required health verification before declaring recovery
  • Connected Argus and Phoenix through the Sentinel Operations Graph
  • Built a unified Sentinel command center for cross-agent incidents
  • Added operator-ready incident reports with evidence-backed decisions
  • Added a seven-stage resilience timeline from healthy state through verification
  • Persisted Phoenix scenarios, approvals, agent runs, and recovery history across restarts
  • Created a guarded live k3s and Chaos Mesh demonstration
  • Created a cluster-free, one-command judge demonstration
  • Added randomized threat and incident generation with deterministic replay
  • Added explicit evidence provenance throughout the dashboards and artifacts
  • Produced auditable JSON and Markdown evidence scorecards
  • Added presentation preflight across the complete platform
  • Documented one clear setup and testing path across all three agents
  • Tracked Build Week changes through issues, commits, tests, and pull requests

What I am most proud of is that the platform does not stop at detection. It follows the problem from evidence, through diagnosis and governance, to recovery and final verification.

What I learned

The biggest thing I learned is that autonomous infrastructure is not mainly an action-execution problem. It is an evidence, verification, and governance problem.

An agent should not act simply because it can. It needs to understand the causal chain, know the potential blast radius, distinguish observation from simulation, explain its reasoning, respect policy boundaries, and prove the outcome afterward.

I also learned that security and reliability should not remain isolated disciplines. A runtime threat and a service failure can affect the same dependency chain. Connecting both perspectives produces better operational decisions than either system can make alone.

Synthetic testing is much more valuable when it does not end with log collection. The real opportunity is connecting a simulated customer failure to telemetry, dependencies, root-cause analysis, remediation, and verification.

Finally, I learned how important it is to be honest in a technical demonstration. I believe the platform becomes more credible, not less, when it clearly distinguishes replayed, simulated, and live evidence and refuses to claim measurements it did not collect.

What's next for argus-k8s

My next step is to take the platform from a strong Kubernetes implementation into a broader autonomous resilience system.

I want to add:

  • Multi-cluster security and resilience coordination
  • More cloud and workload adapters
  • Broader synthetic customer-journey modeling
  • More Chaos Mesh fault types and recovery strategies
  • Longer availability and recovery-objective benchmarks
  • Policy learning from previous operator decisions
  • Expanded predictive-healing memory
  • Stronger causal analysis across metrics, logs, traces, and network flows
  • Multi-tenant governance and approval policies
  • Integrations with incident-management and engineering workflows
  • Production validation across larger and more diverse environments

Built With

Share this project:

Updates