Kafka Agent Run Supervisor

Inspiration

AI Agents can execute code, call tools, and modify files, but operators often have limited visibility after a run begins. If a Runtime freezes, performs suspicious activity, or exits unexpectedly, durable evidence, health monitoring, and a safe recovery mechanism are invaluable resources for operators.

Our Kafka Agent Run Supervisor makes Agent execution observable, auditable, and recoverable.

What it does

Every Agent run produces structured events that are published through Kafka and materialised into a queryable SQLite ledger. From the Supervisor dashboard, operators can:

Monitor active, completed, failed, and cancelled runs.
Inspect correlated event timelines with Kafka topic, partition, and offset information.
Detect suspicious tool activity using deterministic rules.
Identify Runtimes that stop sending heartbeats.
Automatically cancel stalled runs and remove their exact Runtime containers.
Ask a read-only operator chatbot questions about stored run evidence.

The suspicious-activity rules detect and record evidence but do not automatically block an Agent. Automatic enforcement is demonstrated through the reliability path, where a stalled Runtime is cancelled, its labelled container is removed, and the recovery is recorded.

The chatbot can call only six allowlisted, read-only ledger tools. It returns citations linked to stored events, and it never receives log text and executable tools in the same model call. This prevents instructions written into Agent logs from triggering actions.

How we built it

We extended the provided Agent Launchpad rather than replacing its existing Agent lifecycle and Playground.

The React frontend communicates with a Fastify control plane that manages Agent runs and disposable Docker Runtime containers. The Runtime wrapper produces heartbeats and tool-activity events, while the control plane publishes lifecycle events and recovery commands.

Kafka carries the ordered event and command streams. Messages are keyed by runId, ensuring that events belonging to the same run remain ordered as the number of partitions and consumers grows.

A SQLite ledger reconciles the Kafka stream into queryable runs, events, and alerts. Event validation, redaction, and idempotency protect the ledger from malformed, sensitive, duplicated, or replayed data.

A watchdog monitors the latest heartbeat recorded for each active run. When a Runtime stops heartbeating, the watchdog records supervisor.stalled and publishes a single run.cancel command. Before removing anything, the control plane verifies the Runtime instance ID and Docker labels. It then removes that exact container, returns the Agent to ready, and records supervisor.recovered.

Deterministic rules inspect stored tool-activity evidence for suspicious behaviour. The dashboard connects every alert to its matched command and triggering event, allowing operators to understand why it was raised.

The entire reviewer path runs locally using Docker, Docker Compose, Kafka, SQLite, Codex CLI, Ollama, and qwen3:8b. It does not require a paid model API or cloud service.

Challenges we ran into

One challenge was recovering stalled Runtimes without producing duplicate cancellation commands or removing the wrong container. We addressed this using idempotent event and command handling, exact Runtime-instance matching, and label verification before cleanup.

Another challenge was connecting Ollama across Windows, WSL, and Docker while keeping the project reproducible. The control plane probes supported Ollama endpoints, and the repository documents the additional setup needed when Ollama runs on Windows while the control plane runs inside WSL.

We also had to ensure that observability did not become another security risk. Event payloads are redacted before reaching Kafka or SQLite, chatbot tools are read-only, and citations are constructed from stored rows rather than generated model text.

Accomplishments that we're proud of

A complete frontend-to-Runtime middleware path with durable, Kafka-backed evidence for every Agent run.
Automatic detection and cleanup of stalled Runtime containers, followed by successful Agent recovery.
Deterministic suspicious-activity alerts and an evidence-grounded operator chatbot with verifiable citations.

What we learned

We learned that reliable Agent infrastructure depends less on adding more prompts and more on establishing trustworthy boundaries, durable state, and verifiable evidence.

Kafka allowed us to separate event producers, ledger consumers, suspicious-activity rules, and recovery commands while preserving per-run ordering. We also learned that automated recovery must be designed around idempotency and exact resource identity; otherwise, a recovery system could affect the wrong Runtime or repeat a destructive action.

Lastly, we also learned that an operational chatbot should explain evidence rather than control infrastructure. Keeping it read-only and separating tool selection from untrusted log content provides a safer boundary while still making the ledger easier for operators to investigate.

Current limitations

This is a hackathon-scale proof of concept rather than a production multi-tenant platform.

Suspicious-activity rules detect and record but do not automatically block an Agent.
Only the stalled-Runtime path performs automatic containment and recovery.
Kafka runs as a single KRaft node with replication factor 1.
The platform does not provide user identity, RBAC, tenant isolation, or a hardened multi-tenant sandbox.
The local qwen3:8b chatbot may respond slowly or add unsupported wording, although its tools and citations remain grounded in stored ledger data.

These were deliberate trade-offs to keep the three-day project focused, reproducible, and demonstrable on one laptop.

What's next

With more development time, we would add:

An approval-gated remediation Agent that proposes containment actions while keeping the operator in control.
Multi-broker Kafka scaling and ledger replay tooling, while preserving the existing event contracts, per-run ordering, and idempotency.
Alert acknowledgement, per-Agent budgets, and additional sandbox telemetry for network egress and filesystem activity.