Cue | Devpost

Inspiration

Networking events are chaotic, high-pressure environments.

We wanted to build a networking superpower: an assistant that quietly understands who the user is speaking with and provides useful context without interrupting the conversation.

Cue combines computer vision, real-time speech transcription, web research, and generative AI into a discreet heads-up display for professional networking.

What it does

Cue acts as a personal heads-up display for conversations.

Through the Meta Quest headset, Cue captures the wearer’s field of view, and the networking assistant is begins through a trigger phrase activated by the user. Cue then captures an image through the Meta Quest headset, segments the visible faces, and identifies the person the user is speaking with.

Simultaneously, a companion Raspberry Pi equipped with Bluetooth mics captures the conversation. This audio is transcribed in real-time, pulling out key entities and topics. Cue then uses the visual identity with the transcribed context to instantly search the web for relevant background information - such as recent projects, mutual interests, or professional history - and displays those insights directly in your AR view.

How it works

Cue runs as two halves that talk over a private LAN, with no cloud in the identification path.

Seeing. The Quest captures a passthrough camera frame and POSTs the raw JPEG to a laptop server. InsightFace's SCRFD detector returns a bounding box and five facial landmarks per face; those landmarks align the crop, and an ArcFace model turns each face into a 512-dimensional embedding. Identification is one dot product against every enrolled photo; cosine similarity, taking the best score per person, and reporting a name only above a 0.45 threshold. Below that, the system deliberately says nothing: a wrong name on a judge is worse than no name.

Placing. The server returns the bounding box with each name. Unity converts that box into a camera ray, raycasts it against the Quest's environment depth mesh, and anchors a world-space caption at the resulting point. Between frames, faces keep their identity by bounding-box overlap rather than re-embedding, so the caption stays locked to the right head while only a budgeted number of faces are re-embedded per frame.

Hearing. A Raspberry Pi with a DJI Mic Mini streams audio to AssemblyAI and posts finalized transcript chunks to the server. A trigger phrase opens a conversation session; an end phrase closes it and discards the transcript. The Pi never hardcodes an address; the server broadcasts its presence on UDP every second, and the Pi listens for it, so the system survives the venue's Wi-Fi handing out a new IP. Knowing. Research is precomputed, not scraped mid-conversation. A Claude agent loop, given Browserbase's Search and Fetch APIs plus a real cloud browser for JavaScript-heavy pages, researches each attendee and writes two artifacts: concise_research, three to five caption-sized bullets, and verbose_research, every page it read kept whole and unsummarised. The concise version feeds the heads-up display; the verbose version plus the live transcript feed talking-point generation, which refreshes every few seconds as the conversation moves. For LinkedIn: the one source Browserbase can't read past the auth wall: a local Playwright session with a logged-in browser reads the profile instead.

How we built it

The project was built using the following:

Hardware: Meta Quest 3 (passthrough camera, environment depth, world-anchored UI) · Raspberry Pi (audio capture and streaming transcription) · DJI Mic Mini Vision: InsightFace buffalo_l; SCRFD detection plus ArcFace 512-d embeddings, running on the laptop CPU. No face data leaves the machine; recognition needs no API key and no internet. Speech: AssemblyAI streaming transcription from the Pi. Research: Claude (Anthropic) driving Browserbase Search, Fetch, and cloud browser sessions; Playwright for LinkedIn; SQLite for the profile store. Server: FastAPI on the laptop, exposing /id for frames, /transcript for speech, and a conversation event stream the headset polls.

Challenges we ran into

Maintaining low latency
Networking at a live event

Accomplishments that we're proud of

Built an end-to-end wearable networking assistant across a Meta Quest, Raspberry Pi, and cloud AI services
Created identified-person captions in the wearer’s physical environment
Implemented real-time speech transcription from a portable microphone setup

What we learned

We learned that a useful AI assistant is not just about generating a clever response. Timing, trust, and presentation matter just as much.

What's next for htn

Our next steps are to make Cue more reliable, private, and broadly useful:

Improve multi-person tracking and speaker attribution
Add user-controlled preferences for how much information appears in the headset
Expand beyond professional networking to conferences, classrooms, mentorship, and recruiting