ClashGPT

The help page provides easy to understand instructions to quickly start using ClashGPT to analyse your gameplay
Users can connect and choose which style of coaching they receive; from encouraging to getting drilled on their blunders.
Set up the game, start the match and then ClashGPT will immediately start to look for gameplay to analyse.
Sunny gives uplifting and optimistic support as you fight, referencing the current state of the board to show you how you are winning.
ClashGPT provides relevant feedback to the current state of the game, ensuring you stay focused and make the best choices.
As opposed to Sunny, Volt brings a completely different energy to your battles, and ensures you are always ready to win.
Seeing the progress of your efforts, Volt hypes you and your game decisions.
Another image of Volt and his hyping nature.

Inspiration

With the increasing levels, evolutions, and heroes, Clash Royale has become a more pay to win game ever. With ClashGPT, beginners can truly learn the basics and master the game through the help of an AI coach, personalized to your desired feedback.

What it does

ClashGPT records your screen and gathers data such as tower hp and time remaining. Using that data, a request is sent to Gemini to provide quick feedback and suggestions for the game moving forwards. Through ElevenLabs, a personalized trainer audio is used to convert the text to speech. With 5 different types of coaches to choose from, ClashGPT allows users with various preferences to learn more Clash Royale skills.

How we built it

A screenshot is taken every 15s of the user's Clash Royale game and specific features such as the time remaining and hp bars are identified using OCR and open CV. The data is sent to Gemini which is prompted to respond in a particular tone and style based on the coach type selected. Then the response is sent to ElevenLabs which plays the audio in real-time to provide live feedback throughout the game.

Challenges we ran into

Clash Royale is a complicated game with many cards and effects. No model has been developed that is able to identify the current board state and all troops played. After trying to only focus on Hog 2.6, training a model to only identify those 8 cards or using open CV to identify the cards based on size, placement, and movement, the accuracy proved to be too low to be used and we were forced to pivot to other forms of data instead.

Accomplishments that we're proud of

Being able to gather data from the game and have feedback that can help beginners learn what is initially a very complicated game.

What we learned

Learned how to apply OCR to read images and use colour and percentages to determine the hp remaining in towers.

What's next for ClashGPT

To continue improving the model for only hog 2.6 so that players who feel they have a solid foundation of the basics can move on to receving feedback practising against hog 2.6, with proper troop and board identification so that players can master countering one of the most commonly played decks.