optically configured display
full set up

Inspiration

At events like Cal Hacks, you meet hundreds of people. But too often, you run into someone weeks later and have that awkward realization that you can't recall their name, their project, or the last conversation you had with them. We wanted to build an "external memory bank" designed to maintain the bonds of connection. Because the best connections don't happen when we meet more people, they happen when we make the most of the people we meet.

What it does

Memoir is an AR smart glass. There are no buttons or touch sensors to activate. A camera on the glasses spots who you're looking at, the local backend works out who they are and what you know about them, and a tiny OLED screen within your peripheral vision gives you immediate talking points.

Once a conversation starts, Memoir automatically records it. When you walk away, it works out who said what by recognizing your voice and lip-syncing the other person. It then uses Gemini to extract important facts, such as majors, hobbies, or startup ideas, and saves those facts to their profile for the next time you meet.

How we built it

We used a store bought sunglasses as the frame (shout out Walgreens), housing an ESP32-S3 microcontroller, a 128x64 OLED display (SSD1306), and a Logitech C270 webcam that acts as both the camera and microphone.

For our software pipeline, every camera frame runs through SCRFD for face detection, passes a quality gate, and generates an ArcFace faceprint embedding. This embedding is then matched against a TigerData Postgres database using pgvector in under 20ms. To handle hands-free recording, we wait for a face to stay in view for 5 seconds while Silero Voice Activity Detection (VAD) detects speech. Finally, we filter out the wearer's voice using WeSpeaker voiceprints, lip-sync the remaining audio using mouth-opening metrics from the face tracker, and pass the segments to Gemini for fact extraction.

Challenges we ran into

Because we decided to remove the manual touch sensor to make it fully hands-free, we had to solve the incredibly difficult problem of knowing exactly when a conversation starts and who is talking. This required building a shared microsecond clock to perfectly synchronize our audio and video streams so our lip-syncing algorithm could accurately measure mouth openness per frame. In addition, packaging the electronics required fighting through 3D print tolerances, and we spent hours debugging serial communication conflicts such as PermissionError on COM ports between the Arduino and our Python backend, or over Wi-Fi integration.

Another huge issue we encountered was building the actual AR component itself. Since we started this project really late with no prep beforehand, we had no supplies. Thus, we were forced to DIY everything (fun fact: if you break apart our glasses, you can see that we used EVERYTHING to build them. From stickers, to used forks and cardboard, you can find these items stuffed in our glasses compartment to fit everything together). We went to local dollar stores, Walgreen, and even an anarchist hacker group (shoutout Noisebridge) to seek materials. We ended up buying perscription lenses, mirrors, and a clip board to use to shape into our lenses. In the end, we had to simplify last minute because our original design wasn't working, using only the mirror instead.

Accomplishments that we're proud of

We're incredibly proud of getting such a complex local ML pipeline running smoothly within the hackathon time limit. By chaining SCRFD, ArcFace, Silero VAD, WeSpeaker, and Gemini, the system processes frames in ~18 milliseconds.

What we learned

• Synchronizing multimodal AI agents (audio and video) on a microsecond clock. • State management for face tracking and continuous hardware-to-Python serial communication. • Hardware preparation, 3D print tolerancing, and the necessity of having fallback components. • The possibilities of AR!

What's next for Memoir

We hope to refine our hardware and make the lenses more reliable since using the proper convex lense can better help the image to be displayed. Our immediate next step is also to design a custom PCB to replace the bulky prototype wiring. We also plan to transition the ESP32 to stream data fully wirelessly over Wi-Fi, completely untethering the glasses from the laptop to make them a true standalone wearable.