NightFettle

NightFettle is the AgentOps night shift: it health-checks AI agents, lets Codex repair degrading code in an isolated copy, benchmark-gates every change, and delivers a morning report.

Comment

Inspiration

Many Teams all over the world are deploying more AI agents every month: support bots, internal copilots, research assistants, and workflow automations. But after launch, those agents quietly degrade. Tool APIs change, prompts accumulate contradictory instructions, retry loops inflate latency, secrets leak into logs, and success rates drift down. Traditional software has CI, health checks, and Dependabot. AI agents still lack an equivalent maintenance layer.

I built NightFettle to be that layer: the night shift that keeps every agent in fine fettle. (And I believe NightFettle will become one part of the future infrastructure for agents.)

What it does

NightFettle is an open-source, framework-agnostic AgentOps workflow. It closes the loop between observability and repair:

  1. Diagnose — NightFettle reads agent manifests and trace logs, then computes tool success, prompt size, latency, estimated cost, error rate, and security findings.
  2. Isolate — it copies the selected agent into a disposable working directory, so live source code is never edited during diagnosis.
  3. Repair with Codex — Codex reads the source, traces, findings, and acceptance criteria, then produces a concrete code change and a reviewable diff.
  4. Benchmark-gate — machine-checkable tests decide whether the repair passes. A failed repair is discarded. A passing repair only reaches the real agent when the operator explicitly adds --apply.
  5. Report — the Next.js dashboard presents the fleet's health, repair status, measured trace-replay results, and projected impact as a morning report.

The demo includes four deliberately degraded agents covering stale tool schemas, retry storms, prompt bloat, prompt injection, and secret leakage.

How I built it

The core is a TypeScript CLI running on Node.js. Each agent has a manifest, source files, trace fixtures, and benchmark rules. The CLI loads those artifacts, derives health findings, creates an isolated working copy, invokes Codex, prints the Codex Session ID and diff, and runs the benchmark gate before any optional merge.

The web application is built with Next.js 16, React 19, and Tailwind CSS 4. It reads the same checked-in demo data as the CLI, keeping the command-line diagnosis and dashboard consistent. Vercel hosts the public dashboard.

The demo scenarios span LangChain, LangGraph, the OpenAI Agents SDK, and a custom Node.js agent to show that NightFettle is not tied to one framework.

How I used Codex and GPT-5.6

Codex with GPT-5.6 is the repair engine, not a chat widget added to the side of the product. NightFettle hands Codex a scoped working copy, the observed failure, and a machine-checkable definition of “fixed.” Codex must inspect and edit the engineering artifact itself; NightFettle then independently verifies the result.

I also used Codex while building and hardening NightFettle. Real repair sessions exposed weak benchmark criteria, transient-file noise, Windows sandbox conflicts, and the need to distinguish measured fixture replay from projections. Those sessions are documented in the repository README with their Codex Session IDs.

One representative repair updated a LangChain billing agent from the stale invoice_id field to invoiceId. All benchmark checks passed, and fixture trace replay moved from 10/14 to 14/14 successful tool calls. In a security scenario, NightFettle rejected Codex's first attempt because a hard-coded password remained, discarded the working copy, and accepted a later repair only after every security check passed.

Challenges I ran into

The hardest problem was making autonomous code repair safe and demonstrable. A plausible-looking diff is not proof of improvement, so every scenario needed narrow, machine-checkable acceptance criteria. I also had to isolate Codex from the original agent, produce clean diffs, reject partial repairs, and make --apply the only path that can write back.

On Windows, antivirus software conflicted with the native Codex sandbox. I added an explicit compatibility flag while preserving NightFettle's own working-copy isolation. Deploying the monorepo also required configuring Vercel so the web build could read demo data outside the web directory.

Accomplishments that I'am proud of

  • A real end-to-end loop: trace diagnosis → Codex repair → diff → benchmark gate → optional apply.
  • Failed or incomplete repairs are rejected and the original source remains untouched.
  • Five documented Codex repair sessions with reproducible demo agents and Session IDs.
  • One shared data model powering both the CLI and the public morning-report dashboard.
  • Honest result labeling: trace-replay measurements are separated from benchmark-goal projections.

What I learned

Agent observability is only useful if it leads to a controlled action. The most valuable pattern was not “let an AI edit code,” but observe → isolate → repair → verify → require explicit approval. Codex becomes much more useful when it receives a precise failure and an executable definition of success.

What's next for NightFettle

Next I want to add live trace connectors, scheduled night-shift runs, user-defined evaluation suites, GitHub pull-request delivery, notifications, and a persistent history of repairs. The long-term goal is an open maintenance layer that works across every agent framework and gives small teams the same self-improving operations loop as a much larger platform.

Built With

Share this project:

Updates