CommitGym

• 

  environment-lab

• 

  evaluations.png

• 

  evaluations

• 

  lab

• 

  lab

Inspiration

Agents that run real operations fail in a specific way: a tool returns "success" (or a timeout) and the agent believes it. In release engineering, a deploy can time out after it actually committed, a health endpoint can return a cached "healthy", and a migration can write the schema but lose the acknowledgement. I wanted an environment where an agent is graded on what the system actually looks like afterwards, not on what its tools reported.

What CommitGym does

CommitGym simulates a blue-green software release: a web frontend, an API service and a database, with releases v1 and v2. An agent has ten operations: inspect, stage, deploy, health_check, shift_traffic, migrate, verify, retry, rollback and complete. It must take a release to verified v2 or recover safely to v1. Seeded failure scenarios inject an ambiguous deploy timeout, stale health responses and a partially acknowledged schema migration. An independent judge decides whether the final state is safe; the agent cannot declare its own success.

The web workbench lets you run episodes step by step, watch the topology change, inspect each tool result and reward term, replay any saved episode deterministically, and compare policies on held-out seeds.

How it works

• Environment: a Gymnasium environment (CommitGymEnv) with a 22-value observation vector and a Discrete(10) action space. Hidden "true" state is kept separate from what tools report.
• Judge: independent invariants check traffic/schema compatibility, a healthy traffic target, fresh health evidence, resolved operation outcomes and independent verification. complete is accepted only if they all hold.
• Backend: FastAPI and Pydantic, with SQLite persistence of episodes, events and evaluations. Replay recomputes an episode from its seed and actions and checks for an exact match.
• Frontend: React 19, TypeScript and Vite, with an SVG topology, event timeline, reward inspector and evaluation charts.

Reinforcement learning and evaluation

I trained PPO (Stable-Baselines3) on CPU for 300,032 timesteps. My first run without a curriculum failed all 60 validation episodes. A curriculum that starts training from reachable intermediate states fixed exploration. All final evaluation starts from the normal initial state.

The held-out test used 40 unseen seeds × 3 scenarios, 120 episodes per policy. I re-ran it from the saved checkpoint and all 240 per-episode results matched exactly.

PPO earned the higher reward with fewer actions, but it never deployed the new release: it learned that an immediate verified rollback is the cheapest safe outcome. That is a reward-design finding, not a better deployment policy. CommitGym makes that trade-off visible instead of hiding it inside a single success rate, and the UI labels "Deployed v2 · verified" and "Safe rollback · v1 restored" differently everywhere.

Safety and invariant verification

Every operation passes through strict validation. Rejected operations are recorded, never silently executed. Rewards are logged per component: step, invalid, retry, unsafe, progress and terminal. Model output is untrusted: an LLM can only emit one allowlisted operate_simulator tool call with 1–3 enum actions, and the batch stops at the first unexpected result.

Live LLM agent: NVIDIA Nemotron via OpenRouter

NVIDIA Nemotron 3.5 Lightning (nvidia/nemotron-3.5-lightning:free) was integrated through OpenRouter and tested live:

• The model received the real simulator state and produced tool calls. These went through validation, executed in the simulator, and the model chose its next actions from the updated state. Up to 5 decisions per episode.
• In 4 live episodes with the final configuration, it handled real failures correctly: it inspected after a deploy timeout, and it reconciled a partially acknowledged migration. In an earlier test episode it chose a verified rollback.
• None of those episodes reached a safe completion. Twice it called complete while traffic was still drained, and the independent judge rejected it. Twice it ran out of its 5-decision budget.

That is exactly the failure mode CommitGym is built to catch. OpenRouter, not Nebius, was the inference provider.

Technical architecture

React workspace → FastAPI → episode runner → shared CommitGymEnv and independent judge → SQLite. Policies are a deterministic baseline, the PPO checkpoint, live Nemotron, and a separate, clearly labelled offline mock tool adapter. The mock is a local planner that is not an LLM and uses zero tokens. One Docker image serves the API and the built frontend; a Render Blueprint is included.

Challenges and lessons learned

• PPO with no curriculum learned nothing useful (0/60).
• Reward shaping rewarded rollback over deployment, and a single "success rate" hid this until I split the outcomes.
• With Nemotron, hidden reasoning used up the output-token cap before the tool call, so I disabled reasoning for tool calls. A tool-call response is not the same as a correct plan: the judge caught premature completions.

Limitations and future work

• The environment is a finite synthetic state machine with narrow seed variation. These results are not evidence of production generalization.
• PPO's objective needs a deployment incentive or a downtime cost.
• The LLM agent needs better planning within its decision budget.
• Next: more fault families, a downtime penalty, and Nemotron evaluated across many seeds.

Nebius disclosure

Nebius Token Factory and Nebius AI Cloud were not used. Standard onboarding required a payment card, and the organisers confirmed that no card-free participant access or sponsored account was available. The project therefore does not claim Nebius platform integration.

Built With

• docker
• fastapi
• gsap
• gymnasium
• nemotron
• numpy
• nvidia
• openai-python-sdk
• openrouter
• playwright
• ppo
• pydantic
• python
• pytorch
• react
• reinforcement-learning
• render
• sqlite
• stable-baselines3
• tailwind-css
• typescript
• uvicorn
• vite