Inspiration

It's 2 a.m. Your checkout service is down because the database has run out of connections. You ask an AI assistant for help, and it suggests setting the connection pool size to 25 per replica. The calculation looks correct: 3 × 25 + 10 = 85, which is within the database limit of 100.

But there's a problem. Your service can automatically scale up to 6 replicas. Each replica opens its own connection pool, so the actual connection count becomes 6 × 25 + 10 = 160. That's far beyond the database limit. Instead of fixing the outage, the suggested solution makes it worse.

AI can generate answers that sound convincing, but production incidents require more than convincing answers. They require solutions that are verified before anyone acts on them.

That's the idea behind RehearseOps: an incident recovery agent where AI can suggest solutions, but a deterministic safety system has the final say.

What It Does

RehearseOps is an AI-powered incident recovery agent built around a strict separation between decision-making and safety validation.

NVIDIA Nemotron 3 Ultra, running through Nebius Token Factory, analyzes incident evidence, identifies possible root causes, and proposes structured actions with exact parameters. For example, instead of generating a shell command, it proposes an action such as set_pool_size with a pool size of 25.

A separate safety gate, written in plain JavaScript, validates every proposed action. It independently calculates resource limits using trusted facts and the actual parameters supplied by the model. If an action contains an unknown type, unexpected parameter, or unverified fact, it is rejected.

When an action is rejected, the model receives the exact reason and calculation behind the rejection. It then gets another opportunity to revise its proposal. This process can run for up to three rounds, with each step streamed live to the interface.

Once an action passes validation, RehearseOps prepares a rehearsal showing the expected changes, before-and-after values, and a checklist for real-world verification. It can also export a change request. It never executes changes against production systems.

The platform also supports custom logs. Nemotron 3 Nano and a pattern extraction system identify useful facts from uploaded logs. Every extracted value must appear exactly in the original log line used as evidence. If the value cannot be verified against its cited line, it is rejected. Even verified extractions remain untrusted until a human explicitly approves them.

For additional technical context, Tavily searches official documentation during the investigation process. Nano plans the searches, while Ultra uses the retrieved sources and references them through source IDs.

RehearseOps also includes a red-team testing environment with 39 adversarial cases and a live prompt-injection test.

During a real run on our live deployment, Nemotron initially proposed a pool size of 25. The safety gate rejected it because the autoscaler could reach six replicas, resulting in 6 × 25 + 10 = 160 connections against a maximum of 100.

Nemotron received the exact calculation and revised its proposal to a pool size of 15. The revised calculation was 6 × 15 + 10 = 100, which passed validation. The complete run took 12.5 seconds, involved three model calls, and used six Tavily sources.

How We Built It

The backend is built using Node.js 20 and Express, while the frontend uses vanilla JavaScript. We implemented a strict Content Security Policy, self-hosted IBM Plex fonts, four minimal colour themes, and a dark mode.

The interface uses NDJSON streaming so users can follow each investigation, proposal, rejection, and revision as it happens. For AI capabilities, we integrated two NVIDIA Nemotron models through the Nebius Token Factory OpenAI-compatible API.

Nemotron 3 Ultra 550B handles investigation, action proposals, and revisions. Nemotron 3 Nano 30B handles faster tasks such as extracting facts from logs and planning documentation searches. This separation allows us to use a more powerful model for complex reasoning while keeping smaller tasks efficient.

We also built a resilient model integration layer. It can automatically discover available model IDs through the API model catalogue, handle cases where JSON mode is unsupported, remove reasoning traces from generated responses, and retry requests with a higher token budget when reasoning models exceed their initial token limits.

Tavily Search API is integrated directly into the agent workflow. The system also validates source references and removes citations that do not correspond to actual retrieved sources.

The safety engine supports 14 action types and a defined vocabulary of facts with provenance information. Each fact records where it came from, such as a predefined fixture, operator input, pattern extraction, or Nemotron.

We implemented 32 automated tests using locally mocked models, so testing does not consume API credits. The deterministic safety gate also has its own dedicated test suites. The application is deployed on Render using a Blueprint.

Challenges We Faced

One of our most important discoveries came from a mistake in our original safety gate.

The first version calculated connection limits using the current number of replicas. This meant it could approve a pool size that was safe at the moment but dangerous when the service scaled up.

We fixed this by validating against the maximum of the current replica count and the configured autoscaler ceiling. We also included manual scaling scenarios because manually increasing replicas does not prevent the autoscaler from scaling further.

Another major challenge came from our red-team testing. During the first run, a malicious proposal used an object as its action type with a specially crafted toString method. This caused problems inside our gate itself.

We fixed the issue and added it as a permanent regression test to ensure that untrusted model output cannot exploit JavaScript object behaviour.

We also discovered that manually written verification checklists could become inconsistent with the actual proposed action. For example, an action might delete data after 14 days while its checklist incorrectly verified a 30-day retention period.

We redesigned the system so verification checks are generated directly from the action parameters rather than maintained separately.

Finally, different model IDs appeared in display names and API identifiers. We addressed this by implementing automatic model catalogue discovery and resolution.

Accomplishments We're Proud Of

We successfully built a real block-and-revise workflow using live Nemotron 3 Ultra rather than relying on a scripted demonstration.

Across 31 unsafe adversarial proposals, the safety gate produced zero false approvals. It also approved all eight safe control cases, demonstrating that it does not simply reject everything.

During the live prompt-injection test, we supplied an attack message instructing the system to override its rules and set the pool size to 500. The model produced zero influenced proposals, and no unsafe actions were approved.

We also tested a deliberately compliant mock model to simulate what would happen if the AI followed the malicious instruction. Even in that situation, the deterministic safety gate rejected the unsafe action.

Another achievement was designing an interface specifically for incident response. We wanted something calm, clear, and easy to understand, especially when engineers are dealing with stressful production situations.

What We Learned

The biggest lesson was that AI models are excellent at proposing solutions, but they should not be treated as the final authority.

We also found that providing the model with an exact explanation of why an action failed works better than simply asking it to be more careful.

For example, telling the model that 6 × 25 + 10 = 160 and that this exceeds the connection limit of 100 gives it a concrete problem to solve.

Another important lesson was the value of evidence provenance. Every important number should have a traceable source. A value extracted from a log should never automatically become a trusted operational fact.

Most importantly, we learned that safety should not depend on how intelligently a model reasons. It should be enforced independently.

What's Next for RehearseOps

Our next goal is to connect RehearseOps with real infrastructure monitoring systems.

We plan to introduce read-only integrations with Prometheus, Kubernetes HPA, and PostgreSQL pg_stat_activity. These integrations will help automatically collect infrastructure information and convert verified observations into trusted facts.

We also plan to expand the action grammar to support additional recovery operations, including feature flag changes and traffic shifting.

Another direction is deploying the agent through a Nebius Serverless Endpoint to improve scalability and make the system easier to integrate with other environments.

Try It

Live Demo: https://rehearseops.onrender.com

GitHub Repository: https://github.com/JiteshJain123/rehearseops

The project is open source under the MIT License. To explore the application:

Open Integrations and select Verify live call to test the model connection.
Go to Incident Room and select Run recovery agent for the Database Connection Exhaustion incident.
To test custom log processing, open Use Your Own Logs and select Load Example Logs.
Visit the Red-Team Lab and select Run Live Test to see how the system handles adversarial instructions.

The application is hosted on Render's free tier, so the first load may take up to a minute.

Honest Limitations

The four bundled incidents are synthetic examples. The rehearsal engine validates calculations and constraints using trusted facts, but it is not a complete production infrastructure simulator.

The AI's diagnosis can also be incorrect, even when its proposed actions pass the safety checks.

Most importantly, an approved action means that it is safe to proceed with rehearsal based on the available facts. It does not mean the action is automatically safe to execute in a production environment.