BlinkedIn

Detailed View of a Conversation
Ray-Ban Meta Classes on User

Inspiration

Imagine walking into a networking event... blindfolded. For the 285 million people worldwide who are blind or visually impaired, that’s not far from reality. At social events that rely heavily on visual cues, over 56 percent of visually impaired individuals report challenges simply navigating or recognizing who’s around them. This motivates BlinkedIn - an AI assistant to make professional networking more inclusive, restoring independence, confidence, and connection to those who’ve long been left out of the room.

What it does

BlinkedIn transforms Meta Ray-Ban smart glasses into a personal social assistant: the glasses capture a snapshot of the person in front of the user and record their facial characteristics for future recognition. But BlinkedIn does not stop at appearances - it extracts insights from the current dialogue, saves it, and retrieves it for topic recommendation for future conversations. And connection can only be meaningful if it last: After the conversation, BlinkedIn gives users a call, providing them with a brief summary, feedback on networking and assistance to automatically send a follow-up email via voice commands.

How we built it

We used a virtual camera to transmit snapshots captured by Ray-Ban to Gemini, asking it to describe the person in front to the visually impaired user. The image then goes to DeepFace, a deep-learning model which embeds photos into 4096-D vectors. These vectors, along with the conversation transcript, are saved into our database. Insights are continuously extracted from the conversation, giving the user real-time recommendations on possible topics in the conversation. In future meetups, BlinkedIn can match the person's face with conversation history, reminding the users of past common themes. Finally, after each conversation, BlinkedIn also passes the data onto its voice agent, built with Vapi and ElevenLabs, to give user a call to summarize the conversation and offer to send a follow-up email automatically.

Challenges we ran into

Navigating the huge repetoir of AI tools and agents and familiarising ourselves with them in very short periods of time posed a challenge to our team. Additionally, developing a multithreaded, database-integrated system to support multiple users presented technical hurdles in the early stages. Finally, synchronizing workflows and incorporating the code written by different team members also required large amounts of efforts

What we learned

We learned to combine multiple AI agents that work on different types of data in a coherent pipeline. Besides, we can appreciate the importance of prompting techniques: we learnt to write effective prompts for structured output and human-like interaction. We also learned to optimize our project at every stage to meet industry standards of interaction time.

What's next?

We hope to integrate sentimental analysis into the system, analyzing the opponent's facial expressions and predicting probable emotions to paint a clearer picture of the surrounding word for visually impaired people. Feedback on the wording and intonation of the user can also be integrated during the feedback for more educational value.