1201-SF17 ThatsMyJam

Inspiration

We love music, and we love AR. We wanted to combine both using contextual intelligence from a headset — letting AI understand the world around you and turn everyday objects into expressive musical tools. Out of that came a simple question: What if anything you pick up could become an instrument? That idea became That’s My Jam: a playful, immersive way to explore your environment through sound.

What it does

Pick up any object — a banana, a cup, a plastic bottle — and it instantly becomes a musical instrument. Our system: 1. Identifies the object using an AI model 2. Generates a sound personality that matches its tone, feel, and vibe 3. Lets you play it using gesture-based controls and a full musical scale Shake a banana to get slap bass. Tap a cup for warm lofi Rhodes. Pluck a clothespin for pizzicato strings. Every object becomes a new musical experience.

How we built it

We built the front end as an APC app on Quest using Unity. We built a node backend that provided an endpoint to run an image through an LLM for identification and detailed description. Then we connected the two parts, enabling the user on the Quest to identify an object, which sends the image to the endpoint. Then the app receives details on the object and its music. When the user moves their hand holding the object, the app plays different sounds that fit that object.

Challenges we ran into

At midnight, we finally connected the front end and backend… right before the computer we were using stopped sending POST requests for no explainable reason. After two hours of debugging, switching machines magically fixed everything. We also struggled with reliable hand tracking while holding arbitrary objects, and balancing gesture detection with usability. And writing these descriptions with a sleep-deprived brain is always a challenge.

Accomplishments that we're proud of

It’s genuinely fun. We found ourselves wandering IKEA for an hour turning everything into music long after we got our 45-second demo footage. We built something that feels magical, playful, and genuinely shippable with a bit more polish. We’re proud of how complete the experience feels despite the time pressure..

What we learned

• Pun-based names can actually turn into real product names
• When we lock in, we can build fast
• Apparently this building has spontaneous parties at all hours
• And yes — AI can make anything more fun