Wingman

Inspiration

Career fairs are a memory test. You walk an aisle of two hundred booths, and by the time you reach a recruiter you have forgotten what the company does, what they are hiring for, and which of your own projects is worth bringing up. Everyone ends up reading their phone in line, which looks exactly as unprofessional as it feels. We wanted the prep to happen in the two seconds between seeing a banner and stepping up to the table, with nothing in your hands.

What it does:

Wingman runs on the Meta Ray-Ban Display glasses. You walk around normally, but when you look at a company logo or banner, the glasses camera streams to our backend, a vision gate decides whether a real employer banner is in view, and an identify step actually names the company.

The lens shows what the company does, what they are hiring for, and university-recruiting facts, in bullets. A second page is a personal pitch written from your resume against what this employer actually does, so you receive a tailored pitch, connecting your experience directly to the roles the company is hiring for.

Before the fair, you can also paste a list of the companies attending, or drop a screenshot into the app, and every listed company gets researched and stored ahead of time. The user will still receive personalized information, but they can edit any description if they want to.

How we built it

Glasses + iPhone: a Swift app on the Meta Wearables Device Access Toolkit. One device session carries the HEVC camera stream up and the lens renders down. Frames are sampled once a second, downscaled, and sent over a WebSocket, so the backend never judges a booth the wearer has already left. The app also utilizes Clerk sign-in, resume upload, a live mirror of what the lens is showing, a feed of every gate decision with the exact prompt and model response, and the description editor.
Cortex backend: Node and Fastify on EC2 behind CloudFront. A scene gate runs Opus 5 on each frame to identify the relevant company.

Challenges we ran into

Latency vs. cost.: Every frame through a vision model is extremely slow and expensive. After experimentation, we determined that Opus 5 was the model with the best balance of both speed and accuracy.
A 600×600 monocular display with no font control: We measured how many characters fit per row on the real lens and built the row budget into both the renderer and the prompts.
Meta's display recording uses the same camera we stream from, so capturing the demo took some experimentation.

Accomplishments that we're proud of

We're proud to augment Meta Glasses to build a tool relevant to college students. We're proud that we were able to solve the aforementioned technical difficulties, as this is a relatively underexplored area of software.

What we learned

Ship the fallback first. A company list is only useful until you see a booth that is not on it. The live research path, which searches the web and writes a brief on the spot, is what makes Wingman usable at a real fair instead of a demo.
Design for a tiny display. The lens fits only about 38 characters per row and 8 rows. Every prompt, card, and renderer in the system is built around that budget, and we measured it on the actual hardware through testing, instead of guessing.
Vision models need gating. Sending every frame to a model is slow and expensive. Most of the reliability came from deciding when not to call it: constant checks, cooldowns, and backoff after a miss.

What's next for Wingman

Post-booth notes. Say a few words after leaving a booth and Wingman stores them with the company brief and the recruiter's name.
Follow-up list. At the end of the fair, one page with every company you talked to, their application deadlines, and links to the roles from the briefs.
Live badge reading. Recruiter names from name tags, so the pitch page can address the person in front of you.