Inspiration

Could an AI agent play Clash Royale using only what a human sees and the same clicks a human makes? Ever wondered how GPT Sol matches up to Claude Sonnet? Screw Evals, this is the ultimate proving ground for agents!!

Clash Royale combines visual understanding, resource management, timing, and constantly changing strategy, making it a fun challenge for an AI agent that can actually take actions, not just suggest them.

What it does

crankBot watches the game, chooses a card and placement, and turns that decision into clicks.

Our Mac frontend lets you select a screen region containing your mirrored iPhone, calibrate card slots and arena zones, and start the agent. A separate version connects to an emulator orchestrator through configurable screenshot, livestream, and click endpoints.

The Mac version tracks destroyed enemy towers to unlock additional deployment areas. It also measures screenshot-to-action latency and uses approximate movement predictions to adjust placements while troops move during inference.

How we built it

We built the application in Python 3.10 using Factory’s Droid SDK for visual reasoning and decision generation.

We divided the arena into a labeled grid and mapped card slots to normalized coordinates. Droid receives an annotated screenshot and returns a structured JSON decision. Our controller validates the decision and translates it into mouse clicks.

The local frontend uses HTML, CSS, JavaScript, and SVG, served through aiohttp. Screen capture uses MSS, image processing uses Pillow, and macOS mouse control uses Quartz through PyObjC.

We used Mana to develop, iterate, test, and package the project.

Challenges we faced

Latency was the biggest challenge: the game keeps moving while the model thinks. We reduced image sizes, cached overlays, shortened prompts, added model and reasoning-effort controls, and implemented background session warming.

Coordinate mapping was another challenge, especially with Retina scaling and different phone aspect ratios. An interactive calibration interface made those differences manageable.

We also had to handle changing deployment rules after tower destruction, stale frontend state, macOS permissions, and reliable human takeover.

What we learned

An agent needs more than good reasoning. It needs a reliable connection between perception, decisions, and actions.

We learned to separate model judgment from deterministic execution, measure latency explicitly, and test the controller independently of the model. A simulated desktop and game let us validate behavior without requiring a physical device for every change.

Current progress and next steps

The Mac build passes 27 simulated checks, and the orchestrator build passes 28 checks on Python 3.10. These validate software behavior, not competitive gameplay performance.

Next, we want to benchmark live matches, improve motion tracking and card recognition, and validate deployment boundaries and control reliability on real devices.