Inspiration
What it does
Using the Meta Ray Bans to interact with Gemini, we are able to create a live hands-free tutoring session utilizing the built in speakers, microphone, and camera.
How we built it
Integration with the Meta Ray Bans required a phone relay where we utilized Swift to create a mobile app to transmit the prompt to a MacBook, then the response back to the glasses. We also developed a front-end with JavaScript, HTML, and CSS which displays the live feed, project details, and item information. Gemini and ElevenLabs API calls are built with Python.
Challenges we ran into
We frequently got rate-limited in the testing of Gemini. In order to solve this, we optimized our prompt engineering to allow for the most efficient usage of our token limits.
The setup for the prompts from the glasses also became hyper specific, leading to natural language that we expected the tutor to respond to being ignored, leading to generalization of the model's expectations.
Accomplishments that we're proud of
The voice isolation software persisted in loud environments, allowing for the full tutoring setup to operate in those same conditions.
What we learned
The development of a mobile app using Swift and Xcode was a first for all of us. To use that in combination with the Meta SDK and the Meta glasses was all new information that worked out for us.
Log in or sign up for Devpost to join the conversation.