Inspiration

We wanted to see if we could recreate state-of-the-art smart-glasses behavior ourselves in a weekend: real-time, multimodal, on consumer hardware. No hand-waving, just running code.
A last-minute approval for Snap glasses shifted our plans and sparked the goal: replicate the feel of Meta Ray-Bans using off-the-shelf tools, then push latency down with Groq until it felt instant.
The sponsor tech buffet — Studio Lens, Groq, MCP — made this the perfect lab to stress-test a full multimodal stack: vision ticks, tiny rolling memory, wake-word loops, and tool calls that actually do something.

Curiosity first, everything else second: by building the pipeline end-to-end, we proved to ourselves how far modern tooling can go when you keep timestamps tight and outputs structured — and only then noticed how useful it is in the real world.

What it does

Say “hey jarvis” to start a session; get glanceable on-glasses text and a spoken reply without touching your phone.
Every few seconds, Jarvis fuses a fresh visual frame with live speech to surface quick emotion/context hints and a concise, conversational response.
A tiny rolling memory (last 5 moments) keeps the thread coherent, enabling continuous conversation and simple moment bookmarking for follow-ups.
Uses surroundings awareness to tailor suggestions and trigger lightweight actions (e.g., mark a moment, prep a follow-up). Privacy defaults to ephemeral processing with opt-in saves.

How we built it

🛠️ How we built it (end-to-end)

Wake phrase “hey jarvis” opens a session and primes capture.
Every 3 seconds the glasses capture a frame, encode it as base64 JPEG with a timestamp, and ship it to our Python FastAPI endpoint.
A VLM analyzes each frame and returns compact JSON per tick.
We keep a small memory using a stack/queue of size 5; on each new frame the oldest is popped. Frame JSON is merged with Groq-based analysis into a single, stable JSON state.
That state is sent to our final API, which generates the conversational, context-aware response and streams it back; the loop continues until the termination prompt.

Challenges we ran into

Getting comfortable with Studio Lens data flows and tools: wiring vision events, structured outputs, and guardrails so downstream code could trust the schema.
On-glasses UX: crisp, readable text sizing; aligning activation and termination prompts so they feel instant instead of laggy.
Surroundings detection: initial runs frequently reported no recognizable environment; we tuned capture cadence and fallback logic to improve reliability.
Continuous conversation: early builds ended after one full loop; we reworked the session state so it keeps listening and responding seamlessly after each exchange.

Accomplishments that we're proud of

In 36 hours, we assembled a working wearable assistant that feels close to the latest smart-glasses experiences using available sponsor tech.

Approved last-minute for Snap glasses, pivoted fast, and still shipped an end-to-end demo.

Built something technically ambitious that’s also genuinely fun to use.

What we learned

Decent proficiency with Lens Studio and reading docs.
How to stitch VLM vision, speech-to-text/text-to-speech, and fast reasoning into one steady flow.
The unglamorous truths: latency budgets, tiny memories, and session management make or break wearables.

What's next for JARVIS

Scale up with MCP actions: navigation by voice, add calendar tasks, even place a call, all from “hey jarvis.”
On-device models, better speaker diarization, and personal calibration for more stable cues.
Longer sessions: thermal/battery tuning and a richer but still glanceable HUD.