FaceTag

Example Output (1)

Inspiration

As students and passionate learners FaceTag sounded like a super fun idea to tackle. We are frequently going to events, and networking with countless people. We found that the outcome heavily relies on the quality of conversation you have and what you can take away from it. So the ability to see who you should talk to at an event, and review your conversations after the fact is key. We also love to talk, debate, and learn so an in-screen agent that automatically shows you definitions, gives you context, and ideas in general is incredibly cool.

What it does

FaceTag is an app on the Meta Ray-Ban glasses. It captures what you see, what you hear, and what you know and turns it into actionable conversation insights, a personal agent, and a CRM. Think a personal assistant that is always with you.

How we built it

We built FaceTag with three core functions. The app was built in Swift using the Meta Wearables Device Access Toolkit. Our core feature, face recognition, was built using a SCRFD-500M model to find faces and MobileFaceNet to turn each one into an embedding, which we compare against a known database of people in your CRM.

Then as you speak, the conversation is recorded by the glasses, transcribed with Muse, and fed into notes and conversation cues:

Person A: "...and this summer I was an ML intern at Haystack Collective"

Glasses: "Haystack Collective is an investment research platform for asset managers and allocators."

Finally we built Jarvis, an AI agent powered by Fetch.ai to answer any of the users questions.

User: "Where is the nearest Starbucks to me?"

Glasses: "The nearest Starbucks is 0.2 miles away, about a 4 minute walk."

Challenges we ran into

The three biggest problems we ran into were information transfer, conversation inference, and speed. Getting the glasses to produce quick, reliable and genuinely helpful results proved difficult. The actual amount of data we could transfer from the glasses to phone was limited so face recognition was difficult. We also had trouble with speech inference, it was hard to know when to give the user recommendations, context and definitions, and making sure those are delivered in a satisfying user experience.

Accomplishments that we're proud of

On demand quick answers
Real face recognition on a phone, from glasses footage
Speaker attribution with no calibration

What's next for FaceTag

Potential options in the future could be to extend FaceTag to all networking events. Whenever you go to new networking events, people have the ability to see the event beforehand and then already know who they want to connect with. It could also be extended into other fields, like understanding people's emotions for those who find reading emotions difficult.