LifeLens

A privacy-first wearable agent that turns what you see and say into timely next steps for health, work, and relationships—with every action under your control.

Comment

Inspiration

Everyday decisions often start with something right in front of us, yet getting help means stopping to search, type, or switch apps. LifeLens explores an assistant that notices a user-authorized moment and offers a useful next step without demanding constant attention. Our first concrete scenario is understanding a meal and deciding whether to record it.

What it does

LifeLens is an Everyday Agent prototype with two distinct experiences:

  • Structured-moment agent demo: meal, conversation, evening, and movement fixtures exercise the Strands agent's reasoning and confirmation-gated proposals. These scenarios demonstrate the interaction model; they are not evidence of live wearable sensing, health-account integrations, or working route navigation.
  • Live camera and photo experience: a foreground browser session detects scene changes and sends a reduced photo for cloud analysis after explicit consent. It returns food estimates as calorie ranges with uncertainty, supports contextual text questions and speech where the platform provides it, and saves a meal record locally only after confirmation. The browser also offers an optional activity-equivalent calculation based on user-entered weight; it is an estimate, not medical guidance.
  • Vuzix native client: an Android Camera2/HUD client calls the live analysis and question APIs, supports optional system speech recognition/TTS, and records confirmed meals locally. Its signed build is available for device testing. Physical M400/M4000 validation and Korean speech-service compatibility remain pending.
  • Weather work in progress: location-consented weather lookup has KMA and explicitly labeled Open-Meteo fallback code. KMA credentials are not configured, and recent checks of the deployed weather route returned an error. We do not present weather or safer-route guidance as a completed live feature.

Try the live camera/photo experience or the original structured-moment demo.

How we built it

The original agent backend uses Python, the AWS Strands Agents SDK, Amazon Bedrock, FastAPI, and Pydantic. Typed moment input and structured output produce observations, uncertainty, and proposed actions. Strands tool calling powers the confirmation-gated action layer; unconfirmed tool calls cannot mutate memory. This path has been tested with structured meal input against Bedrock.

The new live perception path is separate: React/Next.js and TypeScript or the native Android client call server-side live endpoints, which forward authorized photos and contextual questions to an isolated AWS Lambda using Amazon Nova through the Bedrock Converse API. This path calls Bedrock directly; it does not currently run through Strands or automatically feed the Strands action tools. Confirmed live meal records use the client's local storage. Codex assisted implementation and testing.

Architecture and data handling

Structured agent path: typed moment or fixture -> validation -> Strands + Bedrock -> structured proposal -> user confirmation -> permitted action.

Live path: foreground session + photo consent -> reduced JPEG -> server proxy -> Lambda + Bedrock Nova -> observation or contextual answer -> HUD -> optional, confirmed local meal record.

The original MomentEvent contract rejects raw media. The live path intentionally has a different consent boundary: reduced photos leave the device for cloud analysis, and recognized question text plus current context can also be sent. LifeLens does not write those photos to application storage. This is not a claim that processing is entirely on-device. Speech recognition may use the platform's recognition service. Location is separately consented for weather/place queries. Stopping or leaving the live session stops capture; the native client invalidates late callbacks. AWS credentials stay on the server.

Challenges and results

The main challenge was making the product useful without overstating what a camera, model, or prototype can know. We retain uncertainty, suppress repeated scene notices, require confirmation for records, and keep fixture demonstrations separate from actual inputs. The structured agent policy disallows unsupported claims about feelings, honesty, personality, or medical diagnosis.

Validation includes automated web/backend tests, Android scene-policy unit tests, Android lint, a signed release build, and APK signature verification. An HTTP smoke test using a real food photo returned a food observation through the deployed live endpoint. That verifies the API flow, not glasses performance. We have not measured physical-device latency, battery life, or thermal behavior.

What we learned

Privacy claims must describe the actual data path. A local-observation contract and an opt-in cloud-photo feature cannot share a blanket claim that no images leave the device. Useful agents also need honest failure states: unavailable weather must not become invented guidance, and a successful build must not be described as a tested wearable product.

What's next

Validate the Vuzix experience on physical hardware, resolve weather deployment and credential setup, connect live observations to the Strands proposal layer with explicit confirmation, and evaluate interruption burden and usefulness. Meta Ray-Ban and Samsung adapters, iPhone-to-glasses relay, local vision inference, and AgentCore-backed memory are future work rather than completed integrations. The submitted video and earlier gallery images document the earlier prototype; this description reflects the September 13 update.

Built With

Share this project:

Updates

Submission history