Agent Blackbox

Inspiration

AI agents are running code, editing files, and making decisions — but they're a black box. When something goes wrong (or goes right), you have no idea what actually happened, in what order, and why. There's no replay, no audit trail, no way to debug after the fact.

Existing observability tools are built for web services and cloud infra — not for local terminal-based AI agents. If you're running Claude, a Python agent, or any autonomous CLI tool, you're flying blind. We wanted something that felt like a flight data recorder: always on, always silent, always trustworthy.

---

What it does

Agent Blackbox is a local flight recorder for AI agent sessions. You prefix any command with bb start and instantly get:

• A full timeline replay — every file create, edit, delete, terminal command, and output, in order
• A live streaming dashboard via WebSocket so you can watch a session unfold in real time
• AI-generated summaries with confidence score, root cause, files changed, commands run, and suggested next steps
• Screenshots captured every 5 seconds during the session
• File diffs — not just "this file changed" but exactly what changed, line by line

The dashboard shows all past sessions, a runtime trend chart, success rate, failure count, and average duration at a glance. Sessions are searchable and filterable.

The replay player lets you step through every event with keyboard shortcuts. The right panel shows the AI-generated summary, root cause, files changed, and suggested actions.

The bb CLI is a globally installable npm package — nothing to configure, no provider to sign up for, no API key needed. It works with any command:

bb start claude          # Claude CLI, fully interactive + recorded
bb start python bot.py   # Any Python agent
bb start npm run build   # Even non-AI workflows

---

How we built it

The core challenge was making recording completely transparent. We use node-pty to spawn the child process inside a pseudoterminal (PTY) — so the wrapped CLI thinks it's running in a real terminal. Colors, interactive prompts, keyboard shortcuts, cursor movement: all preserved. The user never notices the recording layer.

The PTY stream is split: one copy goes straight to the user's terminal, another feeds the capture buffer. A chokidar file watcher tracks changes in the working directory. PowerShell captures desktop screenshots every 5 seconds on Windows.

All events are streamed in real time to a self-contained Express + WebSocket backend that also serves the pre-built React frontend as static files — one port (4242), no separate dev server needed at runtime. Sessions are stored as plain JSON in ~/.blackbox/sessions/ — no database, no cloud dependency.

The bb command itself auto-starts the server in a detached background process, opens the browser, wraps the target command, and saves the session on exit. The whole pipeline is invisible.

---

Challenges we ran into

PTY on Windows — Most observability tools punt on Windows. Getting node-pty to compile and run correctly — including raw mode stdin forwarding so interactive CLIs like Claude render properly — required working through VS Build Tools configuration and careful pipe-mode fallback logic.

Transparent recording without breaking the CLI — Claude and similar tools check if stdout is a real TTY and silently change behavior if it's not. Piping was not an option. The PTY approach was the only way to get truly transparent recording.

Live streaming + offline fallback — The recorder needed to work with no dashboard running (filesystem-only mode) and seamlessly upgrade to live WebSocket streaming when the server is available — without posting duplicate events on startup.

Single-package distribution — Bundling the backend, pre-built frontend, and recorder into one globally-installable npm package meant solving a multi-step build pipeline into a single npm run setup-bb command.

---

Accomplishments that we're proud of

• Zero-friction UX — bb start claude just works. The user types the same command they always type, prefixed with bb. That's the entire learning curve.
• True PTY passthrough — Interactive CLIs like Claude work identically inside bb as they do standalone. No color loss, no broken prompts, no UX degradation.
• Provider-agnostic by design — No Gemini SDK, no Anthropic SDK, no OpenAI SDK. It wraps the process at the OS level. It works with any CLI that will ever exist.
• Self-contained package — The entire product — backend, frontend, recorder — ships as one npm package. No Docker, no cloud accounts, no .env files.
• Real-time replay UI — The dashboard feels polished: runtime trend chart, live WebSocket indicator, step-through replay with keyboard shortcuts, confidence-scored AI summaries.

---

What we learned

• PTY (pseudoterminal) is the right abstraction for transparent process wrapping — pipes will always break interactive CLIs in subtle ways.
• Packaging a full-stack app as a single CLI tool is very achievable in Node.js when you commit to serving the pre-built frontend as static files from the same Express server.
• Observability tools are only useful if they're invisible — the moment a user has to think about the recording layer, you've already failed.
• node-pty native compilation on Windows is painful but solvable; for distribution, node-pty-prebuilt-multiarch is the right path.

---

What's next for Agent Blackbox

• npm publish — Ship blackbox-bb publicly with pre-built binaries so end users need zero build tools
• macOS/Linux screenshots — screencapture on macOS, scrot on Linux
• Session diff comparison — Compare two runs of the same agent side by side
• MCP server integration — Expose session data as an MCP tool so agents can introspect their own history mid-run
• LLM-powered summaries — Use actual captured output for richer root cause analysis
• Team sharing — Optional cloud sync so teams can share session replays

Built With

• chokidar
• express.js
• multer
• node-pty
• node.js
• npm
• powershell
• react
• vanilla-css
• vite
• websocket