Midnight Commander

Inspiration

When a production service breaks at 3:00 AM, on-call engineers are forced to drag themselves out of bed, open a laptop, and sift through dashboards while customers are failing to check out. Worse, surface-level alerts often mislead: a downstream feature timing out in the logs might actually be the victim of a degraded upstream payment provider saturating a shared connection pool.

We asked two questions:

Can an on-call engineer resolve—or safely refuse—an incident from bed over a 30-second phone call without ever opening a laptop?
When the AI's first hypothesis is wrong and the engineer declines it and fixes the issue themselves, can the system automatically observe what the engineer did, verify recovery, and propose the right fix the next time that masked outage happens?

That is why we built SecondShift (Midnight Commander).

What it does

SecondShift is a plug-and-play autonomous incident-response and workflow-learning platform powered by CREAO, Sentry, Photon 2.0, ElevenLabs, and Devin (Cognition):

Plug-and-Play Service Manifest (service-manifest.json): Any engineering team can onboard their service without writing custom agent code by declaring their dependencies, read-only health-probe endpoints, latency thresholds, and allowlisted idempotent mitigations in a single service-manifest.json file.
Real-Time Detection & Deduplication (Sentry): When production requests fail, our backend verifies Sentry's HMAC-SHA256 webhook signature, deduplicates webhook retries and recurring error storms into a single active incident, and assigns the on-call owner.
Evidence-Based Investigation (CREAO Mastermind): Our CREAO investigator agent pulls scoped Sentry errors, structured logs, and distributed traces (/tools/sentry_evidence), runs live read-only dependency probes (/tools/probe) to separate surface symptoms from hidden root causes, and checks unified incident memory (/tools/past_decisions & incidents.md).
Two-Layer Deterministic Policy Gate: The LLM proposes; the backend disposes. Before any action can be offered or executed, the backend cross-checks every cited source_id against SQLite to prevent hallucinated or stale (>15m) evidence, enforces the manifest allowlist, and re-runs live dependency probes right before execution.
Actionable Phone Call from Bed (Photon 2.0 + ElevenLabs): Using Photon 2.0's SIP trunk (sip.photon.codes:5061) and ElevenLabs Conversational AI, SecondShift calls the engineer's phone. The engineer can ask "Explain the evidence", say "Approve" to execute the bounded mitigation from bed, or say "Decline" if the proposal is wrong. If unanswered, it retries and escalates to the backup engineer.
Zero-Friction Learning Loop (watchForEngineerFix + incidents.md): When an engineer says "Decline" on the phone, they don't even have to explain why. SecondShift snapshots the raw probe latencies at that moment (e.g., payments: 1800ms, recommendations: 801ms timeout) and polls the app's configuration in the background. When the engineer switches the payment provider to backup in their console, SecondShift automatically detects the manual fix, verifies recovery, and writes the lesson to SQLite and incidents.md. On the next occurrence of that masked outage, CREAO matches the 1800ms probe reading to incidents.md and immediately proposes switch_payment_provider.
Measured Recovery & Pre-Authorised Runbooks: SecondShift never declares an incident "recovered" from silence. It requires >= 10 passing synthetic checkouts across >= 30 seconds AND >= 10 matching ok checkout spans in Sentry. Verified human interventions automatically draft and evaluate a runbook against 12 fixed and held-out safety/refusal test cases—which the lead engineer can authorize for hands-free auto-remediation on future recurrences.

How we built it

CREAO (Central Investigator Brain): We built a reusable CREAO agent triggered via POST https://agent.creao.ai/api/v1/runs with connected HTTP tools (/tools/service_manifest, /tools/sentry_evidence, /tools/feature_config, /tools/probe, /tools/past_decisions, and /tools/submit_investigation) and attached service-manifest.json in CREAO's Files tab.
Sentry (Errors + Structured Logs + Distributed Tracing): We instrumented our checkout service with @sentry/node v11 using three distinct Sentry products:
1. Error Monitoring (Sentry.captureException + signed alert webhooks),
2. Structured Logs (Sentry.logger.info / Sentry.logger.error with request_id, run_id, dependency, and latency_ms), and
3. Distributed Tracing (Sentry.startSpan for checkout, dependency.recommendations, and dependency.payment). Our backend queries Sentry's Explore API (/api/0/organizations/{org}/events/) across the errors, logs, and spans datasets, and also reports its own operational errors to Sentry (backend/src/monitoring.ts).
Photon 2.0 + ElevenLabs (Voice & SIP Telephony): We provisioned a Photon 2.0 voice-capable number (+13465660558) on a TLS SIP profile (sip.photon.codes:5061 with SHA-256 digest auth and inbound routing to sips:sip.rtc.elevenlabs.io:5061) and connected it to ElevenLabs (eleven_v4_turbo TTS at 1.2x speed + Scribe v2 Realtime ASR). We wrote an automated provisioning script (npm run configure:voice) that reads service-manifest.json and configures the ElevenLabs agent's prompt, dynamic variables, noise/VAD settings, and webhook tools via PATCH /v1/convai/agents/:id.
Backend & Incident Desk (Node.js 24, TypeScript, node:sqlite, Express): A zero-external-DB backend using Node's built-in node:sqlite (WAL mode) storing incidents, evidence, timeline_events, proposed_actions, action_executions, verification_samples, decision_feedback, runbook_versions, calls, and retrospectives, paired with a live dark-mode Incident Desk dashboard (/dashboard) and interactive architecture specification deck (/architecture).
Devin by Cognition (AI Software Engineer): We built, debugged, and verified the system alongside Devin—writing 138 automated assertions across 5 self-test suites, inspecting raw SIP packet traces and ElevenLabs turn-metrics JSON, and refactoring the codebase into a manifest-driven plug-and-play engine.

Challenges we ran into

How Sentry Observability Shaped Our Build (The Dedupe Bug & Missing Spans): Early on, our recovery verifier and recurring-alert counter weren't seeing repeated checkout timeouts in Sentry even though synthetic traffic was failing every 2 seconds. By inspecting Sentry's Explore view, we realized @sentry/node's default Dedupe integration was silently dropping consecutive identical timeout exceptions! Disabling Dedupe in demo-app/src/instrument.ts immediately restored full telemetry visibility. Similarly, Sentry's trace spans showed us that when recommendations.lookup times out at 800ms, the payment.charge span never executes—proving why absent payment errors in logs do not mean payments are healthy, which led directly to our independent live probe design.
Disambiguating Masked Shared-Pool Failures: Under providerDegradation, the primary payment provider slows down (1800ms) and saturates a connection pool shared with the recommendations client, causing recommendations to time out at 800ms. On Incident 1, logs only show recommendation timeouts. We had to design both a backend pre-execution safety net (checkApplicability blocking disable_recommendations when payment probe latency exceeds 1000ms) and an evidence-snapshot comparison loop so CREAO learns from the engineer's provider switch.
Hardening Live Phone Calls Against Hackathon Venue Noise & SIP Timeouts: During our first live phone test in a noisy room, ambient chatter triggered ElevenLabs' barge-in detector (interrupted: true in the turn metrics), cutting off the agent mid-sentence after tool calls. Using Devin, we inspected the conversation turn-metrics JSON from the ElevenLabs API and patched our agent config with vad.background_voice_detection: true, turn_eagerness: "patient", speculative_turn: false, disable_first_message_interruptions: true, interruption_mode: "disable_during_tool_and_turn", and ASR keyword boosting. We also discovered that ElevenLabs' POST /v1/convai/sip-trunk/outbound-call can return a 60s HTTP timeout after a call is already answered and approved, which initially triggered a duplicate backup call; we fixed voice.ts to check live call state before firing backup retries.

Accomplishments that we're proud of

True "Approve from Bed" Experience: Going from a real checkout outage → Sentry telemetry → CREAO investigation → Photon + ElevenLabs phone call → spoken "Approve" → idempotent flag flip → 10 verified checkouts + 10 Sentry traces → auto-updated incidents.md in under 2 minutes without touching Slack or a terminal.
Learning from "No" Without Asking the Engineer to Explain: An engineer can simply say "Decline" on the phone, hang up, and switch the payment provider in their own console. SecondShift snapshots the raw probe latencies at decline time, detects the config change automatically (watchForEngineerFix), verifies recovery, and teaches CREAO to propose switch_payment_provider next time.
138 Automated Checks Across 5 Self-Test Suites: Every safety invariant—webhook retry deduplication, anti-hallucination source_id checks, stale-evidence rejection, slow-payment safety net,unenrolled caller rejection, and 12 fixed + held-out runbook evaluation cases—is verified in npm run selftest.

What we learned

Observability is an active control-loop input, not just a passive dashboard: Combining historical Sentry spans/logs with fresh read-only dependency probes lets an agent reason about what didn't run as well as what failed.
Qualitative evidence status beats invented LLM confidence scores: Forcing the investigator to classify hypotheses strictly as supported, inconclusive, or contradicted with persisted source_id citations—while keeping execution permissions in deterministic backend code—makes AI incident response trustworthy.
Workflow learning is data capture, not fine-tuning: Recording { diagnosis_then, raw_probe_snapshot, engineer_decision, verified_resolution } in SQLite and incidents.md is immediately inspectable, auditable in git, and requires zero model retraining.

What's next for SecondShift (Midnight Commander)

Multi-Channel Post-Call Follow-Ups on Photon 2.0: Sending an automatic iMessage / SMS summary with the incidents.md diff and retrospective link via Photon 2.0 as soon as the 30-second post-call verification window completes.
Drop-In SDKs (@secondshift/express, @secondshift/fastapi): Packaging the /config, /probe/:dependency, and /actions/:action contract as a 5-line middleware so teams can generate service-manifest.json automatically from their code.
Multi-Service Dependency Graphs: Extending service-manifest.json across microservice topologies so CREAO can trace cascading timeouts across multiple upstream services.