Inspiration

For many people, rhythm games provide an engaging combination of heart-pumping beats and skillful hand-eye-ear coordination. The skill ceiling in many of these games is extremely high, and those that are new to the genre or wish to improve may find themselves intimidated or not knowing how to overcome their current technical wall. While some of the post-game analytics can be useful, players may not know how to fully analyze them to know where and how to improve.

Typically, players repeat songs over and over many times to figure out what's wrong in a “brute force” fashion, but this is often time consuming and doesn’t leave players with a proper understanding of what needs to change. Additionally, players may injure themselves due to bad habits or be given advice by experts that may have worked for them, but lack a personalization that would best benefit the aspiring player.

To tackle this issue, we made Musical Spartans, an AI powered rhythm game trainer that combines smart glasses, AI, and a companion app to provide players personalized insights into their gameplay and how to best improve.

What it does

Musical Spartans is an AI-powered rhythm game training system that uses Unity, Mentra Live AI smart glasses, and a React-based companion app to provide personalized feedback based on gameplay performance.

Players begin by playing a rhythm game developed in Unity. During each session, the game tracks performance metrics such as:

  • Song information and BPM (beats per minute)
  • Player score and overall accuracy
  • Correctly hit and missed notes
  • Note timing and performance consistency

After completing a song, the gameplay statistics are exported into a CSV file. A QR code allows the player's performance data to be retrieved through the companion application. Using Mentra Live smart glasses, players can scan the QR code to initiate the transfer of their gameplay results. The React application then uses this information to generate a performance summary and provide AI-powered coaching recommendations. Players can also interact with an AI chatbot to ask questions about their performance and receive suggestions for improving their rhythm and timing.

How we built it

Our project combines multiple technologies in order to make a connection between rhythm games, AI, and smart glasses.

Unity - Rhythm Game Development

We used Unity to develop the rhythm game, implementing gameplay mechanics involving note timing, player inputs, scoring, and accuracy tracking. The games were based on osu!mania (a 4, 6, or 7-key rhythm game) and Taiko no Tatsujin (a drum-based rhythm game). The game records performance statistics during each session, which can then be exported and accessed via QR code for further analysis.

React - Companion Application

We used React to develop the companion application that communicates with our backend and AI coaching system. The application serves as the central interface where players can review their gameplay statistics, receive performance summaries, and interact with an AI coaching chatbot.

We also worked on parsing CSV files so that the data exported from the rhythm game can be structured into an organized format for analysis and presentation.

Mentra Live - Smart Glasses Integration

We used the Mentra Live SDK to integrate smart glasses into our rhythm game training system. The glasses allow players to scan QR codes associated with their gameplay sessions, providing a convenient way to retrieve performance data through the companion application.

Microsoft Azure - Backend

Microsoft Azure hosts the Rhythm Backend as an ASP.NET Core service through Azure App Service. The backend provides session registration, authenticated CSV uploads, QR-based read access, attempt listing, downloads, and Gemini coaching requests.

Azure Blob Storage privately stores session metadata and gameplay files, while App Service environment variables protect deployment configuration and API credentials.

AI - Rhythm Game Coach

The backend sends gameplay CSV data to Gemini through a secure Azure-hosted API. Gemini analyzes available timing, accuracy, BPM, note-density, and consistency metrics, then returns structured coaching with a focus area, actionable practice drill, and next target.

The system avoids inventing unavailable measurements and treats blank values conservatively.

Challenges we ran into

One of our biggest challenges was connecting multiple technologies into a working system. Unity, React, Microsoft Azure, and the Mentra Live SDK use different environments, so we needed to establish a reliable way for information to pass between them.

This also required us to consider compatibility between different programming languages, frameworks, and APIs. Since Unity primarily uses C# while our companion application uses React, we needed to establish a standardized method of communication between the different components.

Another challenge was developing a specific format of data. We first needed to determine the data format that would be exported from the Unity game. Integration of exporting data also relied on finding efficient methods.

Our group decided that one of the optimal methods for this was the use of QR codes, which can be scanned with the smart glasses. We also had to design a reliable data format shared across Unity, React, and the backend.

We chose CSV because it is easy to export, validate, transfer, and inspect. QR codes provided a convenient way for the React application to retrieve the correct session without manually entering an identifier.

Additionally, we had to consider how AI could provide genuinely useful coaching advice instead of simply repeating statistics already available to the player. We had to give it context regarding rhythm games so it could better understand common mistakes, potential causes of injury, and pitfalls players may encounter.

Accomplishments that we're proud of

We are proud of creating a project that combines different types of technologies to solve a common problem for rhythm gamers. We are also proud of the steps we made along the way to making this project possible, such as:

  • Developing a rhythm game using Unity with gameplay performance tracking.
  • Building a React-based companion application for viewing statistics and interacting with AI.
  • Establishing a CSV-based format for communicating gameplay statistics.
  • Exploring QR-code-based data transfer between the game and companion application.
  • Working with the Mentra Live SDK to incorporate wearable smart glasses into the experience.
  • Designing an AI coaching workflow that turns gameplay statistics into personalized recommendations.
  • Setting up a stable data flow between the RhythmEngine, backend, Mentra Live glasses, and companion application.

What we learned

We learned more about game development in Unity, particularly how to track player interactions and collect meaningful performance metrics.

We also gained experience with React development, including parsing CSV files, managing structured data, building user interfaces, and designing an application that communicates with other components through APIs.

Working with the Mentra Live SDK introduced us to wearable application development and the challenges of integrating smart glasses into a larger software system.

One of our biggest takeaways was the importance of data standardization and communication between platforms. Even when individual components work correctly, connecting them requires careful planning and coordination.

We also learned that useful AI feedback depends heavily on the quality of the information provided. Detailed timing and accuracy data can make coaching recommendations much more meaningful than a simple overall score.

We also learned how to set up a Microsoft Azure Web App and transmit data using an API between the game, backend, and React companion application.

What's next for Musical Spartans

In the future, we would like to:

  • Improve AI coaching: Provide more detailed feedback based on note timing, accuracy patterns, and difficult song sections.
  • Expand smart glasses integration: Allow players to receive coaching tips and performance summaries directly through other types of smart glasses.
  • Personalized practice recommendations: Suggest specific songs, BPM ranges, or exercises based on each player's weaknesses.
  • Predictive Analysis: Determine whether a certain song or game may be too difficult or complex for a player based on their current skill level.
  • Heuristic Analysis: Understand common patterns across popular rhythm-game types and use that to make tailored practice suggestions.

Built With

Share this project:

Updates

Submission history