Fargate Spot Autopilot

Automatic fallback from Fargate Spot to on-demand Fargate, with gradual recovery inside the same ECS service without taking over desiredCount.

Comment

Inspiration

The idea came from looking at a production AWS bill and seeing ECS at the top. Fargate Spot can reduce compute cost significantly, but an ECS service does not automatically fall back to regular Fargate when Spot capacity becomes unavailable. The public AWS feature request for this behavior is still marked “Not planned.”

I wanted to build the smallest possible controller that closes this gap: something that plugs into the existing autoscaling setup instead of competing with it.

What it does

Fargate Spot Autopilot adds automatic in-place fallback and gradual recovery to an existing autoscaled ECS service.

It observes ECS events and scheduled ticks, then runs a deterministic state machine:

  • NORMAL: use the configured mix of Spot and on-demand tasks
  • FALLBACK: move partially or fully to on-demand Fargate when Spot becomes unstable
  • RECOVERY: introduce real application Spot tasks as canaries and gradually restore the target mix

The controller never writes desiredCount. Service Auto Scaling remains its only owner. The controller changes only the ECS capacity provider strategy.

How I built it

The project is written in TypeScript and uses AWS CDK, Lambda, DynamoDB, EventBridge, the AWS SDK v3, and Amazon ECS.

The policy is implemented as a pure deterministic state machine shared by two environments:

  1. a local simulator that runs without AWS credentials or cost;
  2. a real AWS adapter that reconciles the same decisions against ECS.

The simulator demonstrates interruption-burst detection, immediate placement-failure fallback, gradual canary recovery, failed recovery, and autoscaling during recovery. I also ran an end-to-end AWS smoke test that verified real ECS deployments and task placements.

I built the project with Codex using GPT-5.6 Sol at high reasoning effort. Codex built the core controller, state machine, local simulator, policy tests, CDK infrastructure, and AWS smoke harness from my specifications. I made the architecture and product decisions, reviewed the implementation, and later used an independent AI review and real AWS validation to find and fix issues before submission.

Challenges

One challenge was preserving Service Auto Scaling ownership of desiredCount while still expressing exact task counts through ECS capacity provider strategies. Plain weight ratios are rounded unpredictably for small services, so the controller uses base and weight together and recomputes the strategy against the live desired count.

Another challenge appeared only during the real AWS test: changing the capacity provider strategy required a forced ECS deployment before tasks were replaced. The smoke test exposed that assumption and led to a corrected rollout implementation.

What I learned

The agent is only as fast as its feedback loop. Because the policy was a pure state machine, Codex could run the full behavior suite locally with no AWS credentials and no deploy cycle. Every property that made the code testable also made it agent-friendly.

On the ECS side: reliable Spot recovery has less to do with capacity than with rollout control. It has to cooperate with whatever autoscaling the service already has, not compete with it.

Also: preparing the demo took about ten times longer than building the thing. I'm told this is normal.

What's next

The next step is making the controller work well for fleets of 100+ tasks. On a large service, the fixed interruption count that signals trouble for a small fleet can just be normal background churn. The controller needs rate-based thresholds tied to the number of Spot tasks actually running.

Built With

Share this project:

Updates