Cloudfall: an on-call agent that proposes, never acts alone

Nemotron investigates your production incident; a person approves the fix; Cloudfall keeps the record

Comment

Inspiration

Small teams run their own servers, and nobody wants the 3 a.m. pager. AI agents can already read logs and run commands, but giving one shell access to production means trusting the model never to make the wrong move. We wanted the opposite: an agent that is useful at 3 a.m. because the architecture, not the model, decides what it is allowed to change.

What it does

You give Cloudfall an alert. cloudfall agent investigate hands it to an NVIDIA Nemotron model with the team's own operations catalog as tools: Ansible playbooks, each marked read, mutating or destructive.

  • Read operations run and return what the hosts reported
  • Every change runs in Ansible check mode and is recorded as a proposal with its diff
  • No tool approves anything. A person approves; Cloudfall runs the change, verifies it on the host and keeps the decision record
  • cloudfall why later answers what changed, who approved it, and which turn of which model asked for it, with that turn's reasoning

The demo works a real incident. On a Hetzner server, 36G of unrotated shop logs fill the disk and PostgreSQL goes down; the alert only says "PostgreSQL is down". Nemotron 3 Super reads the logs, finds the cause and proposes rotating the logs, then restarting PostgreSQL. Approved by a person, its fix brought PostgreSQL back on the real host, and rerun with verification on a fresh one it took the disk from 100% to 5%.

Try it at https://demo.cloudfall.dev: pick any of four Nemotron models, watch it work turn by turn with its raw reasoning and tool calls, approve the fix yourself, then ask the record why.

How we built it

  • Nemotron on Nebius Token Factory through its OpenAI-compatible API, with tool calling. Nemotron 3 Super is the default model
  • Our own small tool loop inside Cloudfall (Python, open source), no agent framework. Tools are generated from the operations catalog, so the model only sees what the team declared
  • Reasoning on the record. Each turn's reasoning_content, the tool calls exactly as the model wrote them, the Token Factory response id and x-request-id, and the token counts are kept in an investigation record next to every decision
  • Replay against a live model. --replay answers the model's calls from the output recorded on the real broken host, so any model can be tried on a real incident again and again without a server. The hosted demo and our evaluation both run that way
  • The gate is Cloudfall's decision record: check-mode diff, approval by a person, verify run, all as JSON with schemas
  • The demo is a small Starlette app streaming the CLI's events to the page, deployed with Cloudfall itself to a Hetzner server behind Cloudflare, with per-visitor and daily token limits

Nebius and NVIDIA usage

  • Runtime calls to Nebius Token Factory on every investigation, in the CLI and in the hosted demo
  • Four NVIDIA Nemotron models evaluated: nvidia/nemotron-3-super-120b-a12b (default), nvidia/Nemotron-3-Ultra-550b-a55b, nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B, nvidia/Nemotron-3_5-Lightning
  • Nemotron's reasoning_content and Token Factory's request ids are stored per turn, so every proposed change can be traced to the model's reasoning

Results

Four Nemotron models, ten runs each, on the real broken host:

Model Right fix Proposed dropping the database Tried to approve itself Median tokens
Nemotron 3 Super 10 of 10 0 0 12,724
Nemotron 3 Ultra 8 of 10 0 2 17,235
Nemotron 3 Nano 9 of 10 0 0 21,568
Nemotron 3.5 Lightning 0 of 10 10 of 10 0 54,932

Super was right every time, with the fewest tokens and a median of 15 seconds. With tools that run whatever the model asks, Lightning would have wiped the production database in every run. With Cloudfall, none of the 40 runs changed the host.

Challenges we ran into

  • Our first approved fix failed for real. It wrote a config file before freeing space, and Ansible could not create its temp directory on a full disk. Check mode passed; the verify run caught it and Cloudfall recorded NOT_VERIFIED. We reordered the playbook and the host recovered. This is exactly why the verify step exists
  • Models reach for the shortcut. In an early test, 3 of 4 models appended --yes to a restart command and invented flags. The biggest model, Ultra, tried to approve its own proposals. We removed every approve path from the agent: approval needs a person
  • Our own gate got one verdict wrong. Writing up the real run, we found Cloudfall had recorded a fix as verified although its verify run changed the host again: it read only the last Ansible task's result. We filed it publicly (#76), said so in the proving-run report and fixed it: verdicts now come from the play recap, and a test with a real ansible-playbook keeps it that way
  • A demo that is honest. Our first demo page looked like a simulation. We added the raw reasoning, raw tool calls, request ids, the host output with its recording hash and a downloadable trace, so a judge can check every step

Accomplishments that we're proud of

  • A real incident and a real fix: approved on a real server, it brought PostgreSQL back, and rerun with verification on a fresh one it took the disk from 100% to 5%
  • A 40-run evaluation showing that model choice matters, and that the gate held for all of them
  • Replay: real incidents become regression tests for agents, runnable by anyone with a Token Factory key

What we learned

  • Bigger was more eager, not safer: the largest model was the one that tried to approve itself
  • A small set of declared operations beats a shell. The model is better at choosing from a catalog than at writing commands, and every choice is checkable
  • Keeping the model's raw reasoning next to each change makes post-incident review far easier than a summary

What's next

  • More recorded incidents (certificate expiry, a crashed deploy, memory pressure) as a public benchmark for on-call agents
  • Alert intake from monitoring, so an investigation starts on its own and pages a person only to approve
  • Pilots with small teams and MSPs who run their own servers

What changed since August 26

Cloudfall began in July 2026 as Atlas, a private tool for our own Hetzner servers. By the start of the submission period it could describe a fleet in YAML with JSON Schemas, render an Ansible inventory, observe hosts and audit them against the declared state, and show an operations dashboard, with Ansible roles for our Nginx proxies. It could not deploy, migrate or talk to an agent.

On 2026-09-07 it became the public open-source Cloudfall. Since then, in nine releases (0.1.0 on 2026-09-08 to 0.7.0 on 2026-10-19):

  • Health-gated deploys with rollback, the artifact builder and the Render importer, with the Render-to-Hetzner migration proven live end to end
  • The MCP server and the agent-facing CLI
  • The operator loop: propose in check mode, approve, run, verify
  • The operations catalog and cloudfall why

For this hackathon specifically:

  • cloudfall agent investigate with Nemotron on Token Factory, streaming each turn
  • The investigation record with reasoning, raw tool calls and request ids
  • --replay of recorded incidents against a live model
  • Approval only by a person
  • cloudfall why naming the model, turn and reasoning behind each decision
  • The disk-full example and the 40-run evaluation
  • The hosted demo

Built With

  • ansible
  • cloudflare
  • hetzner-cloud
  • nebius-token-factory
  • nvidia-nemotron
  • python
  • starlette
  • uv
Share this project:

Updates

Submission history