Inspiration
Have you ever listened to a song, and relived an entire night from it? Capturing that feeling, that moment, is exactly what PROJECT_MOMENT is all about. On our phones we separate recognizing songs into an app like Shazam, and taking photos as just a camera app. But these two things serve a similar purpose, to capture a special moment. When we transition to AR glasses, we can rethink these basic apps into having a greater purpose. My project is about capturing the big and little moments with both music and photos.
What it does
Here's the flow:
- User hears cool music in the background
- thumb tap microgesture to recognize song
- Learns about the song and artist
- Takes a photo in that moment, making a cool album cover (a Moment) of what they saw and heard!
The user also has their own gallery of Moments that they can look back on with a full screen viewer, so they can sit back, and rewind!
How we built it
Used microgestures for the input system with this layout:
- Thumb tap(Right hand) to start recording audio
- Pinch to take a photo
- Pinch blocked automatically when UI is open
- Thumb tap(Left hand) to open user gallery
- Pinch to scroll/open the
Used Unity 6 + Meta SDK 78’s PassthroughCameraAccess, I built a pipeline that:
- Captures passthrough frames as raw RGBA buffers
- Converts them into Texture2D images
- Saves them into memory ^^ This was some tough stuff
A full Audio-capture pipeline:
- Samples the mic
- Generates a WAV file
- Sends it to a custom backend deployed on render.com
- Receives identified song + artist from ACRCloud music recognition api
Some other stuff I built:
- A MemoryDatabase storing textures + metadata
- A MemoryLibrary UI that filled entries dynamically
- A FullPhotoViewer to display selected memories
And finally the HUD display to make the whole thing not feel like an app, but an experience you can live your life around without thinking about it.
Challenges we ran into
Too many to count. The biggest challenge for me was having to move my whole thing from unity 2022.3 to unity 6 which broke pretty much everything. It wasn't until the last four days of the project where I finally got it to work and then ended up finding many more mini challenges in between.
But mentally this one was rough. I'm 18 years old and had my first university midterms, while also spending 90% of my day working on this project and making content (shameless plug LakshXR on IG). I'm learning how to do everything from scratch and by myself as well which was something I am very happy to have pushed myself to do.
Accomplishments that we're proud of
Too many to count once again. The most important I could say is that, I did something I genuinely thought was not possible by me. I'm not the best at sticking to a project till the end and heck, build anything half as cool as Moment!
What's next for Project_Moment
Tons of nitty gritty bugs and things to make the experience better of course which I'll be cleaning up this week but other than that, there's the big picture. I'd love to be able to share Moments or look at albums with my friends in an app like this. Doing something like that in person is already fun, but for friends further away that would be truly amazing. And then of course bringing this thing to AR glasses to create true immersion, in your Moments.
Log in or sign up for Devpost to join the conversation.