Meta Rayban Glasses + Gemini

I'm very excited about the AI wearables, Meta glasses have a lot of potential, but their closed and limited functionality sucks, plugging Gemini onto it, is a game changer.

Comment

Inspiration

The Meta Rayban Glasses, with their innovative introduction into the wearable technology market, have shown great promise. However, I noticed their software and AI lacked of functionality due to a closed ecosystem. My project is driven by the goal to fully unlock these glasses' potential by integrating them with WhatsApp (initially) and harnessing the power of Google Gemini, thus transforming them into a highly versatile personal assistant.

What it does

My project transforms the Meta Rayban Glasses from a simple tech novelty into an essential interactive tool by facilitating communication with a WhatsApp bot. This allows me to perform Google searches, store notes in Notion, manage calendar events, analyze images, and interact with prompts—all via text messages. Commands begin with "Hey Meta, send a message to Gemini Glasses," followed by the instruction. The system intelligently processes these commands, whether it's to create a Notion page, schedule a calendar event, or analyze images using Gemini Vision.

How I built it

The project was crafted using the Google Gemini API, Python, and Fast API for the backend. Integration with WhatsApp is achieved through a webhook that interprets the message content to determine the action required and extract the necessary information. For image analysis, images are stored in Redis, with follow-up messages related to "image" triggering Gemini Vision for information extraction. Deployment is managed on Cloud RUN, ensuring scalability and efficiency.

Challenges I ran into

A significant challenge was achieving seamless integration between the glasses and WhatsApp for prompt command execution. A notable limitation was the glasses' inability to read WhatsApp messages longer than 15 words, often causing delays. To overcome the limitations imposed by the closed ecosystem, I'm delving into hacking the glasses' OS and Meta View app, aiming for a direct connection to a custom app or finding an improved interaction method with the glasses.

Accomplishments that I'm proud of

I am proud of devising a system that not only enhances the Meta Rayban Glasses' capabilities but also paves the way for mass adoption of advanced AI wearables. By leveraging the widespread ownership of Meta glasses, I believe I can create software into the glasses that can perform as the Rabbit R1 or Humane Pin, fostering significant adoption and transforming the concept of wearable technology.

What I learned

This project has enriched my expertise in the Google Gemini API, enhanced my understanding of WhatsApp API utilization, and improved my skills in image analysis with Gemini Vision. It has significantly expanded my knowledge of integrating diverse APIs, deploying cloud-based applications, and the potential of elevating wearable technology through sophisticated software enhancements.

What's next for Meta Rayban Glasses + Gemini

The primary focus moving forward is to break free from the glasses' closed ecosystem by hacking into the OS and the Meta View app. This approach will open up a realm of possibilities, allowing for the full utilization of the glasses' remarkable hardware. By creating an open ecosystem, I aim to develop a platform that can rival leading AI wearables, ensuring the Meta Rayban Glasses are not just widely owned but also universally embraced for their enhanced capabilities and versatility.

Built With

Share this project:

Updates

Submission history