-
-
System Architecture
-
Tools and Techniques Used in Implementation
-
Use of Gemini Multimodal Model
Inspiration
Mobile games often reward two very different abilities at the same time: the ability to think strategically and the ability to execute quickly and accurately on a small touch screen. Many people have the first ability but are unfairly limited by the second. A player may know exactly what move should happen next, yet still struggle because of reduced eyesight, limited hand dexterity, tremor, slow reaction time, difficulty with timing-sensitive interactions, or discomfort using small devices. This affects people across ages and backgrounds, including older adults and people with temporary or permanent physical limitations.
That gap led us to build Play My Strategy. We asked a simple question: what if AI could serve as the player’s eyes and hands, while the human remains in control of the intent? With modern multimodal models now able to interpret visual game scenes, and Android tools able to control devices programmatically, we saw an opportunity to rethink mobile gaming as something more accessible, more strategic, and more inclusive. In the long run, we also believe this can create a new style of play for everyone: instead of competing on reflexes alone, players can compete on the quality of their ideas and strategies.
What it does
Play My Strategy lets a user describe how they want to play a game in plain English, then has AI carry out that strategy on their behalf. The user can type their strategy or provide it by voice. After that, the system continuously observes the game screen, interprets the current state, decides what move best matches the user’s stated strategy, and applies that move to the phone.
A key design choice is that we are not asking the model to simply find the optimal move. Instead, we ask it to act the way the user would want to act, based on the strategy they provided. That makes the system personal and intention-driven rather than just automated. In our current prototype, the system can successfully play the popular game 2048, where a user can provide strategies such as keeping the largest tile in one corner, preserving board structure, and avoiding disruptive moves. The result is a new interaction model: the player supplies the plan, and the AI handles perception and action.
How we built it
We designed the system as two cooperating components: a Local Game Driver and a Cloud-based Intelligence Backend.
The Game Driver runs on the user’s machine, such as a laptop. It connects to an Android phone over USB or WiFi using ADB, captures the phone screen, crops the relevant game region, and executes the returned action on the device through taps or swipes. This local component also provides the user interface for entering strategy text or voice input. One important advantage of this design is that it does not require special game-specific apps; the goal is to work with ordinary Android games.
The second component is the Cloud Intelligence Backend, which we deployed on Google Cloud Run as a FastAPI service. The local driver sends the cropped game image and the user’s strategy to this backend. The backend then calls Gemini 3 Flash Preview for multimodal reasoning: it reads the game state from the image, interprets it in light of the user’s instructions, and returns the next move in a structured format. We used OpenCV for local image preprocessing and Firestore to maintain session and move history. Altogether, the system forms a clean pipeline from user intent, to visual understanding, to action execution.
Challenges we ran into
One of the biggest challenges was latency in the interactive game loop. The system must repeatedly capture the screen, interpret the board state, query the cloud model, and execute the returned action, so even small delays can slow the experience. To reduce that overhead, we optimized both local preprocessing and cloud interaction. On the client side, we crop only the game board, resize it before upload, and use lightweight checks to avoid unnecessary cloud calls. On the backend, we use a tightly scoped board-only prompt, strict JSON output, and a bounded thinking budget to reduce reasoning and parsing overhead.
We also had to make the end-to-end perception and control loop robust. Unlike a simulator, our system operates on a live phone, where errors can come from screen misalignment, unreadable crops, invalid model outputs, or moves that do not actually change the board. To improve reliability, we added lightweight safeguards across the pipeline. The client verifies that the phone is connected before starting, compares pre- and post-move board crops to confirm that an action had an effect, and triggers a fallback move when the game appears stuck. On the backend, we enforce a strict request-response schema and validate structured outputs to reduce downstream failures.
Another challenge was making the system easy to try. Although our backend is publicly hosted on Google Cloud Run, the full system still depends on a local Game Driver that users must run on their own machines to connect to and control an Android phone through ADB. That setup requirement creates a practical barrier for broader access. To lower that barrier, we began building a browser-based playground that reuses the same Cloud Run backend without requiring local setup or phone connectivity. The playground is already functional, but because each cloud-hosted inference call incurs a cost, we are currently sharing it selectively rather than opening it broadly to the public.
From the beginning, we also wanted the project to be more than a one-off demo, so extensibility became a design challenge of its own. Our goal was to build a framework that could support additional games, interfaces, and accessibility-oriented use cases over time. To do that, we separated the system into clear components for local control, FastAPI routes, Gemini-based planning, Firestore storage, typed data models, and browser playground support. This modular design makes the system easier to extend, adapt, and maintain as the project grows.
Accomplishments that we're proud of
We are proud that we were able to turn a broad accessibility idea into a working, demoable system in a short amount of time. The prototype successfully plays 2048 end to end: the user provides strategy, the system reads the board, plans the move, and executes it automatically on the device.
We are also proud of the architecture itself. By separating the local execution layer from the cloud intelligence layer, we created a design that is modular and extensible. Even though our current demo focuses on one board game, the overall framework can be expanded incrementally to support more games with relatively small additions on the Game Driver side.
Most importantly, we are proud that the project reframes accessibility in a meaningful way. It does not remove the player from the loop; it preserves the player’s agency. The human still chooses the strategy. The AI simply enables that strategy to be carried out.
What we learned
This project taught us a great deal about practical multimodal AI systems. One major lesson was how much multimodal model capability has improved. We had used Gemini before and were initially unsure whether it would reliably understand a game board, including numbers and their spatial arrangement. In this project, we found that Gemini 3 Flash Preview could interpret structured game scenes much better than we expected, which gave us confidence that this approach can extend beyond a single demo.
We also learned that deployment on Google Cloud Run is much more approachable than it first appears. The process was straightforward, the interface was intuitive, and the ability to manage resources and expose a clean API made it a strong fit for rapid prototyping.
At the same time, we learned that remote API-based LLM inference is still too slow for many real-time interaction tasks. That insight shaped our thinking for the future: if this idea is to scale to faster games and smoother interaction, we will likely need smaller models, fine-tuned models, or local inference paths that reduce response time substantially.
What's next for Play My Strategy
Our next step is to expand beyond 2048 and support a wider range of games with different mechanics and interaction patterns. We also want to experiment with smaller and possibly fine-tuned models to improve speed and make the system more responsive for dynamic gameplay.
Beyond engineering, we want to study the human side more carefully. We plan to run a user study in which participants play different games, describe their strategies, and interact with the system. This will help us answer important questions: Is the model truly following the user’s strategy? How should we measure strategy fidelity? How can we make strategy input easier to write, revise, and refine over time?
Finally, we want to build a broader ecosystem around strategy-driven play. We imagine shared strategy libraries, game-specific leaderboards, and a community where people compare not just scores, but the quality of their strategic thinking. Our long-term goal is simple but ambitious: to make mobile gaming accessible to many more people, and to open up a new way of playing where ideas matter as much as reflexes.
Log in or sign up for Devpost to join the conversation.