What it does
Capture the Square is a simple top-down fighting game, where teams of two fight to control a central area. What makes it unique is that the AI in the game are controlled by a Reinforcement Learning algorithm, which learned how to play the game with no human input. This allows for a much more hands-off and goal-oriented approach to creating AI for games.
How we built it
Our team used the Python library Pygame to handle visuals and game events, and drawing software to create the sprites. We used a Reinforcement Learning library that we've been working on to train the agent. In order to quickly train the agents for repeated testing within the time constraints, data collected from each agent is fed into a single Reinforcement Learning algorithm (rather than one algorithm for each agent), which learns to optimize play for each agent independently. Additionally, the game was coded such that several instances of the game could be run in parallel, further increasing the rate of data collection. The agent observation is a 48x48 RGB image, and the agent outputs WASD actions and an angle in 18 degree intervals.
Challenges we ran into
Reinforcement Learning trains agents to optimize reward gain, however, this can produce unexpected behavior. Several challenges with agent behavior include:
- Agents spawn-killing other teams instead of controlling the center, preventing the team being spawn-killed from learning (implemented "bases", where agents get damaged when too close to an enemy spawn)
- Bow agents learning slower due to difficulty aiming, preventing meaningful interaction with other agents early on (implemented auto aiming)
- Non-humanlike behavior such as spinning and hitting air (currently unresolved, could be resolved with more training and tuning of rewards)
We also experienced difficulties with creating the game. Some challenges include:
- Providing readability despite the low resolution
- Implementing functional arrows on a grid layout
Accomplishments that we're proud of
The game's aesthetics turned out much better than we thought it would, and is surprisingly satisfying to play. Additionally, while the trained agents are not perfect, they are definitely entertaining to watch. Agents also gained a surprising amount of understanding of the game for only being trained for half an hour, some examples include sword agents achieving high hit ratios and bow agents kiting towards the edges of the capture area.
What's next for Capture the Square
Some next steps include increasing map size for more focus on area control, more humanlike movement which could be enforced as an additional reward to optimize, and a league system to create better performing and more robust agents.
Log in or sign up for Devpost to join the conversation.