Night Watchman

Inspiration

I run cron jobs on a small EC2 box — data pipelines, backups. Every monitoring tool tells me they ran. None tells me they actually worked. The classic silent failure: exit code 0, green checkmark, and a truncated or malformed output quietly poisoning everything downstream. Enterprise AI-SRE platforms solve this for companies with full observability stacks; nothing serves the solo builder with five cron jobs on a VPS. I wanted an agent I could trust at 3am — and I believe trust comes from evidence quality, not autonomy.

What it does

Night Watchman checks whether your cron jobs actually did their job — and when they didn't, investigates and brings you evidence, not verdicts.

1. Detect (deterministic, no LLM). An inspector checks every job's output against declared expectations and its own history: existence, size, row count, schema, anomalies vs. prior runs. Every observation carries the raw value and its source path. Uncomputable metrics are labeled unavailable — never silently defaulted.
2. Investigate (GPT-5.6, evidence-only). When something is flagged, deterministic collectors gather log tails, file samples, and history trends — then GPT-5.6 reasons over that evidence package alone. No filesystem access. Every claim must cite a source pointer that exists in the package; a validator rejects invented citations. The dossier states what the evidence shows, suspected areas in probabilistic language, what was not checked, and the decision the human needs to make — with options and risk notes. Never a verdict.
3. Approve-to-execute (Telegram). The dossier arrives on your phone with inline buttons for exactly two bounded, non-destructive actions: quarantine_and_rerun and rerun_only. After execution the inspector runs again and replies with the re-inspection evidence — including flags that remain. The system never claims success; it shows you the post-action evidence and lets you conclude.

How we built it

Codex was the implementing engineer; I was the architect and reviewer. The project's principles — evidence only, missing data labeled as missing, deterministic where possible — plus a hard scope guard were written into AGENTS.md before the first line of code, and Codex enforced them throughout, routing feature ideas into TODO.md instead of code. ~50 tests grew alongside features across small commits: detection layer → investigation layer → Telegram escalation → approve-to-execute, in 4 days. GPT-5.6 investigates at runtime; the safety properties around it (citation validation, schema enforcement, bounded actions) are deterministic code built by Codex. The full day-by-day human–Codex log is in CODEX_LOG.md in the repo.

Challenges we ran into

The best challenge became the demo. During failure-injection testing I truncated the output CSV — destroying its header. The job kept exiting 0 and appending rows without restoring it: a persistent silent failure, invisible to any heartbeat monitor. When I asked Codex to make the job rewrite headers on malformed files, its implementation deliberately preserved non-empty files instead — the right call (a job shouldn't overwrite what might be real data). We adopted the scenario as the core demo. The payoff: GPT-5.6, from history and log contradictions alone, reconstructed that the file had been truncated externally — without ever seeing the job's source code.

What we learned

Writing principles into AGENTS.md before coding turned an AI coding agent from an autocomplete into a colleague that pushes back correctly. And "evidence, not verdicts" survived contact with reality: when a rerun didn't fix the root cause, the system honestly reported the remaining flag — exactly as its own risk note had predicted.

What's next

A generic job registry beyond the two demo types, richer investigation playbooks, and more bounded action types — all currently living in TODO.md, by design.

Built With

• codex
• gpt-5.6
• openai
• python
• sqlite
• telegram