Cloud-RL

Inspiration

Cloud infrastructure is a continuous balancing problem. Reducing cost too aggressively can affect reliability, while improving security can introduce operational overhead or downtime. We wanted to explore whether reinforcement learning agents could learn these trade-offs instead of optimizing for a single objective.

This led us to Cloud-RL, a cloud operations learning environment where agents make decisions across simulated infrastructure while dealing with cost, security, reliability, dependencies, failures, and uncertainty.

Our goal was to build more than a simple RL demonstration. We wanted an environment where an agent's decisions have meaningful consequences and where different learning approaches can be trained, evaluated, and compared under the same conditions.

What it does

Cloud-RL provides a simulated cloud environment containing resources such as VMs, databases, caches, and queues. Each resource has properties including health, cost, utilization, security risk, vulnerabilities, encryption, logging, credentials, and dependency relationships.

During an episode, the infrastructure can experience drift and incidents such as vulnerabilities, exposed resources, stale credentials, degraded health, and dependency failures.

Agents can perform cloud-operation actions such as:

ANALYZE · PATCH · ENCRYPT · ROTATE_CREDS · RESTART · RESIZE_DOWN · STOP · DELETE · NOOP

Every action is validated against the current environment state and safety rules. Dangerous operations can be rejected before execution, allowing agents to learn while operating within defined constraints.

Cloud-RL currently supports two learning approaches:

Math RL uses MaskablePPO to train a structured agent directly from numerical cloud-state observations with invalid-action masking.

LLM RL uses a custom GRPO-style pipeline where a language model generates multiple structured JSON actions for the same cloud state. The candidates are validated, rewarded, compared using group-relative performance, and used to improve the policy.

The platform evaluates agents using metrics such as reward, win rate, cost, security, reliability, savings, safety violations, cascade events, action diversity, JSON validity, and evaluation stability.

How we built it

Cloud-RL was built in Python using Gymnasium, Stable-Baselines3, SB3-Contrib, NumPy, PyTorch, Transformers, and Gradio.

The environment represents cloud infrastructure as a stateful system rather than a collection of independent actions. Resources maintain their own operational state and dependency relationships, while the environment introduces changes and incidents throughout an episode.

Our reward system is multi-objective:

$$ R = w_cR_{cost} + w_sR_{security} +

w_rR_{reliability}

w_aC_{action}

w_fP_{failure} $$

This encourages the agent to balance cost efficiency, security, and service reliability while penalizing unnecessary actions, failures, and cascading incidents.

For the structured RL path, we use MaskablePPO with action masking so the agent does not waste exploration on invalid actions.

For the LLM path, we developed a custom GRPO-style training loop. The model generates a group of candidate actions, each candidate is parsed using a strict JSON schema, validated against the environment, and evaluated using the environment's reward system.

The relative advantage can be represented as:

$$ A_i = \frac{r_i-\mu_r} {\sigma_r+\epsilon} $$

where (r_i) is the reward of a candidate action and (\mu_r,\sigma_r) represent the reward statistics of the candidate group.

We also implemented curriculum progression, dependency-aware failure propagation, safety vetoes, and an immune-inspired memory system that tracks recurring failure patterns and cascade behavior.

A Gradio interface provides training, evaluation, monitoring, and visualization of the resulting metrics.

Challenges we ran into

The biggest challenge was designing a reward system that encouraged useful behavior rather than exploiting a single metric. An agent focused too heavily on cost could learn to shut down resources aggressively, while an agent focused only on security could perform expensive operations without considering reliability.

Another challenge was preventing unsafe exploration. Reinforcement learning depends on experimentation, but unrestricted experimentation is not appropriate for infrastructure operations. This led us to introduce action masking, validation, safety vetoes, and restrictions around critical resources and failure states.

Representing dependencies was another difficult part. A resource cannot always be evaluated independently because actions affecting one resource can impact other connected services. We therefore implemented dependency-aware state transitions and failure propagation.

For the LLM-based agent, we also needed to make natural-language decisions measurable. A strict JSON action format allowed us to parse, validate, score, compare, and evaluate every decision consistently.

Accomplishments that we're proud of

We are proud that Cloud-RL combines structured reinforcement learning and LLM-based reinforcement learning inside the same cloud operations environment.

We built a stateful infrastructure simulator where decisions affect cost, security, reliability, dependencies, and future states rather than producing isolated rewards.

We also made safety a core part of the environment. The agent can propose an action, but the environment determines whether that action is admissible.

Another major accomplishment is the immune-inspired memory system, which allows the environment to retain information about recurring failures and cascade patterns instead of treating every episode as completely independent.

Finally, Cloud-RL provides a complete evaluation layer rather than measuring only training reward. This makes it possible to compare different agents and policies using multiple operational metrics.

What we learned

Cloud-RL taught us that effective agentic decision-making requires much more than generating a correct response.

An agent operating in a dynamic environment needs to understand the current state, consider possible consequences, respect constraints, balance competing objectives, and learn from previous outcomes.

We also learned that reward design is behavior design. The behavior an agent learns depends heavily on what the environment rewards and penalizes.

Building the LLM-RL pipeline also showed us the importance of structured outputs. By enforcing a strict action contract, we were able to turn an LLM's decision into something that could be objectively validated and evaluated.

Most importantly, we learned that reinforcement learning becomes significantly more interesting when the environment itself has memory, dependencies, uncertainty, and consequences.

What's next for Cloud-RL

Our next goal is to expand Cloud-RL into a larger benchmark for AI-driven cloud operations.

We want to introduce more realistic infrastructure scenarios, larger environments, additional RL algorithms, stronger language models, and more complex multi-step tasks.

We also plan to explore richer long-term memory, multi-agent cloud operations, controlled integrations with real cloud services, and standardized benchmarks for comparing RL and LLM-based agents.

Ultimately, we want Cloud-RL to become a platform for training and evaluating AI agents that can make safer, smarter infrastructure decisions before those policies are trusted with increasingly realistic cloud environments.