We've already built an app that allows you to drop memories and stories in the world around you (Marble AR). But we did not have any spatial audio yet and loved the idea to extend it with audio-based AR, as the Bose glasses can provide. Plus, we were fascinated by the SDL automotive API. So we thought what we might do with all of this :-)
What it does
The idea is to let musicians conquer streets with their own music, i.e. to let them drop at certain street sections they can own. If other people drive by, they hear these audio tracks approach (and go by) spatially. If they decide to continue listening, they tap on the Bose glasses to stay tuned into that track. Another idea was to promote concert venues provided by the real-time Bands-in-town API, while you're driving by.
How I built it
What is already working:
- Drop music tracks in the world, by uploading MP3 files
- The tracks are displayed as glowing, rotating music notes
- If you approach them, the track starts playing in spatial audio, via the Bose frames (unfortunately, you cannot hear this in the video, as the sound cannot be recorded in parallel; but it worked well!)
Challenges I ran into
Our biggest challenge was to get the SDL API up and running. The idea was to build a native Unity plugin, as the SDL APIs are only available for native iOS and Android. Although the basic plugin worked, the SDK crashed for whatever reason, if we called it inside Unity. We tried over 8+ hrs to get this fixed, but couldn't make it happen, unfortunately. So the general idea to demo all the stuff while simulating a drive-by and pushing the music tracks to the dashboard display could not be accomplished. But at least we learned a lot on our way :-)
Accomplishments that I'm proud of
The integration of Bose AR and Bandintown went like a breeze. It took a while to get used to the Bose framework, but due to the good documentation, there were no bigger hurdles. The Band-in-town API only provided events nearby, based on GPS locations, but it did not tell us the GPS locations of the actual venues. So we had to hook into a geolocation-based search provider (komoot) to get the actual latitudes and longitudes. Worked.
What I learned
Maybe it would have been better to not try to get the automotive stuff done for hours. But we really wanted to make this happen, so we couldn't let off...
What's next for Drive-by Radio
- Stabilize the stuff and integrate it into the official Marble app, once it is released (we'd need an official band-in-town API key for this, though). Wink.
- Build a working Unity plugin for SDL, and maybe publish it on Github? We think having Unity as an additional appü ecosystem would be an amazing opportunity for SDL, especially as it opens a lot of opportunities for both AR, VR, and gaming (for the kids in the car, maybe, not for the driver :-)
PS: tried to link a Git repo, but Gitlab won’t let me change my visibility settings (at least on my iPad)