Inspiration
The track that immediately drew me in was Fire TV. The premise of the technology was very intriguing to me. I had a lot of TVs and displays lying about that were just collecting dust.
The main drawing point of the Fire Stick for me was the idea of repurposing old tech that would otherwise be unused. I thought I would stretch that attribute into the app idea itself.
I wanted to focus on computer vision, which usually would require extra hardware (such as a camera), but... in keeping with the theme of repurposing, I thought I would make it a point to avoid that. Instead, this app allows you to use your phone (that you likely already have) to act as the camera.
The app is called Fire Beats.
What it does
Fire Beats is a hardware-free rhythm and workout game designed to be used on your Fire TV through the combination of computer vision and WebRTC. This eliminates the need to buy extra equipment and offloads the heavy work onto your phone, which likely will have better processing power than a TV, while also keeping latency minimal.
How It Works
Getting started takes seconds and requires zero configuration by design. Players simply open Fire Beats on their TV and scan an on-screen QR code using their smartphone. This instantly establishes a secure, ultra-low-latency WebRTC connection between both screens.
Once a music track is selected, the player props up their phone, steps back, and steps into the camera frame. As the music plays, dynamic rhythm targets appear in the four corners of the television screen. Using computer vision running locally on the phone, the app tracks the player's upper body and hand positions in real-time. To score points and keep the rhythm alive, players must physically move their arms to smash the generated "fire beats" in perfect time with the music.
Personalization and Scoreboards
I wanted to introduce some personalization and competitive encouragement, so I added a way for players to take a picture of their face and slap it on the avatar on the screen. I also allowed local arcade-like leaderboards so you can compete with your friends and family members. This could be perfect for a party game!
If you have your own music you want to dance to, you can easily upload your own tracks and Fire Beats will automatically generate deterministic beat sequences according to the audio analysis of that track. Perhaps you and your friend want to prove who the better fan of your favorite artist is?
Alexa Voice Control
To avoid obstructing the gameplay screen with buttons, I integrated Alexa voice control to allow players to use voice commands such as play and pause to control a game session while a track is playing. This way the rhythm game sequence will stay focused while still allowing players to pause should they need to.
How we built it
Fire Beats primarily works by using the user's phone browser for its camera and the resulting MediaPipe pose detection. It then sends those pose landmarks through WebRTC. Importantly, the pose inference only happens in the phone browser and Fire Beats never adds the camera stream to the WebRTC connection through a low-latency data channel.
In the Fire TV application itself is where we draw the game canvas and on-screen avatar. It parses the received WebRTC pose data to visualize the user on the TV for player feedback.
The phone then continues to use the WebRTC connection so it can act as a remote, such as for taking avatar pictures and music selection.
The Node.js backend acts as the bridge between the two interfaces. It's here where we decode audio and beat analysis to generate unique game sequences.
Challenges we ran into
I think the hardest challenge was creating the beat / rhythm generation based on the music the user uploads in a way that makes each music track a unique experience for a player without overwhelming them. I ended up making a desktop app for quick prototyping of beat generations just so I can "feel" what's right without ending up drenched in sweat testing physically on my TV.
While WebRTC data channel was an almost perfect solution for the purpose of this app, it still introduced a little bit of inherent latency. So I had to adjust the "hit" detection to be a bit more forgiving without making it too easy.
Accomplishments that we're proud of
Initially I wanted to also stream the camera through the TV but latency and resolution became an immediately noticeable problem. I was glad I found a compromise that turned out better anyway. The avatar representation not only reduces latency and avoids the ugly resolution problem, the resulting aesthetic has more identity and unique style and makes it more of a "game".
What we learned
I learned that I got a Fire Stick to play around and develop with now! Seriously it was such a good investment: it was super cheap and I can use it beyond this hackathon.
I was quite far removed from TV app development since this was my first time, even though I'm an Android developer. But after this experience I realized that there's nothing to worry about. It was a smooth experience overall.
From developing this Fire TV app, I also learned the clear distinctions between TV and mobile apps and the right mindset to use when developing each kind. I'm talking about the 10-foot experience (TVs) vs. the 1-foot experience (phones). Such as how TVs are usually shared and lean-back vs how a phone is personal and lean-forward; as well as what TVs provides that phones just don't, such as the viewing real estate that allowed me to build this rhythm game in the first place.
What's next for Fire Beats
- Using Alexa Skill to preload a track on the TV instead of navigating on the phone.
- More variations in "hitting the beats" to increase difficulty
- More complex audio track analysis, perhaps including lyrics to cue the players into doing certain motions when relevant words show up
Log in or sign up for Devpost to join the conversation.