We thought it would be cool to add an extra level of interactivity to Twitch streaming by integrating Augmented Reality into streams, particularly streams featuring musicians and music to increase fan engagement and music discovery.
What it does
We built 3 scenarios:
- A static flair that a streamer/musician/artist can let float around near the stream, a personal or team logo.
- A physical embodiment of a musician as they stream themselves performing or writing music, controlled using wearable technology (i.e. glove).
- A set of Augmented Reality easter eggs in the stream that can be used to enhance games that the streamer can run with their viewers (i.e. "Guess the song based on an audio snippet and related visuals)
How we built it
We used Unity with Vuforia for AR to build the iOS app Firebase as a database and to send messages back and forth TiVo to source images Twitch Extensions to build the streaming integration
Challenges we ran into
Getting the Twitch Extension up and running e.g. A small obstacle that we overcame eventually was using Twitch’s developer-rig tool (v0.6.0), where to getting it to work requires downgrading to Node 10 from Node 11. Another small obstacle is to loading an image on the Twitch Extension overlay, which turned out we just have to wait a bit for the image to load.
Accomplishments that we're proud of
Creating a link from Twitch to an AR based app, including integration of a wearable device.
What we learned
That Firebase tends to blow up mobile apps.
What’s next for BeyondTwitch
- With streaming becoming more and more popular, it is logical that music album's premium content can move from physical artworks to digital artworks (e.g. AR objects). To prevent "copy-and-pasting" of these digital artworks, Ethereum's ERC-721 non-fungible token contract can be used to verify the authenticity of these AR objects, and also allow fans to trade these digital items and interact with other fans globally on existing ERC-721 markets like OpenSea. So, next is to integrate with Ethereum’s ERC721 Non-Fungible Token contract, so that artists can release limited edition AR items to the audience, where the digital item is verifiable to be authentic and unique.
- Add an AR item where it unlocks a bonus soundtrack, to surprise the fans, and increase engagement.
- Add a leaderboard for further gamify the music discovery experience.