Remember

Inspiration

We constantly lose small but important details: where we put our keys, which bag belongs to us, what was said during a busy moment, or where we last saw an object. Traditional note-taking does not solve this well because it requires us to stop what we are doing and deliberately record everything.

Remember was inspired by a simple question:

What if your glasses could help you remember naturally, from your own point of view?

Instead of recording an endless video or forcing the wearer to organize notes, Remember converts selected moments into compact, searchable memories. The goal is not to remember everything—it is to preserve the details that become useful later.

What it does

Remember is a voice-first memory assistant for Snap Spectacles.

The wearer can look at a scene and say:

“Remember this.”

Remember captures the current view and uses GPT-5.6 to identify relevant objects, actions, locations, spatial relationships, readable text, and uncertainty. It then compresses that information into an evolving memory graph.

Later, the wearer can ask natural questions such as:

• “Remember, where did I put my keys?”
• “What was beside my notebook?”
• “What do you remember about the blue bag?”
• “What happened this morning?”
• “Where did I last see my glasses?”

Remember retrieves the most relevant evidence, generates a concise grounded answer, displays the supporting capture when available, and speaks the answer aloud.

The wearer can also manage memories conversationally:

• Correct or update a memory
• Reinforce an important fact
• Summarize memories by topic or time
• Delete or forget specific memories
• Undo the previous memory action
• Pause all capture with private mode
• Switch between manual and automatic capture

How we built it

Remember is built in Lens Studio for Snap Spectacles using TypeScript.

The experience combines:

• Spectacles’ first-person camera
• VoiceML and ASR for conversational input
• Location modules for optional place context
• Snap’s Remote Service Gateway
• GPT-5.6 for visual understanding and grounded recall
• OpenAI text-to-speech for spoken responses
• Optional Supabase persistence
• A local memory graph for efficient retrieval

GPT-5.6 is used in two main stages.

1. Perception and memory compression

A captured image is passed to GPT-5.6 with a structured prompt. The model returns strict JSON containing:

• A concise summary
• The visible action
• Objects and aliases
• Scene and event context
• Object states and changes
• Spatial relationships
• Readable text
• Evidence strength
• Confidence and uncertainty

This creates a searchable memory rather than an unstructured image archive.

2. Grounded recall

When the wearer asks a question, Remember first performs deterministic local retrieval. It ranks memories using relevance, recency, confidence, importance, reinforcement, lifecycle status, and conversational context.

Only the best evidence is packed into the GPT-5.6 prompt. The answer stage is instructed to distinguish between directly observed, user-stated, and inferred information—and to acknowledge when the evidence is insufficient.

The evolving memory graph

Remember does not treat every capture as an isolated note. Memories are merged into a graph containing:

• Episodes
• Objects
• Locations
• Temporal segments
• Relationships
• State changes
• Supporting evidence

Repeated observations strengthen an existing memory instead of creating endless duplicates. Corrections preserve earlier values while promoting the newest canonical version.

The graph can remain local or optionally sync to Supabase as one compact session record, with selected thumbnails stored separately.

How Codex helped

Codex was used throughout the core engineering workflow to inspect the Lens Studio scene, understand the existing controller, implement features, identify fragile state transitions, and validate changes across the project.

It accelerated work on several complex areas:

• Designing the evolving memory lifecycle
• Grounding generated answers in retrieved evidence
• Resolving references such as “that memory” or “the previous one”
• Reconciling spoken facts with existing visual memories
• Handling confirmations during or immediately after speech
• Preventing cloud-sync race conditions
• Cleaning up thumbnails when memories are deleted
• Adding offline and speech-failure recovery
• Integrating approximate location with safe fallbacks
• Packaging dependencies, sample data, testing instructions, and Git LFS assets

The most important product decisions remained explicit in the implementation rather than being delegated entirely to generation.

Challenges we ran into

Turning images into useful memories

A generic image description is not enough to answer future questions. “A table with several objects” is technically correct but practically useless.

We had to prompt for actionable details: which object was moved, where it was placed, what it was beside, whether text was readable, and which details were uncertain.

Preventing confident but unsupported answers

A memory assistant becomes harmful if it confidently invents where something was left.

We addressed this by separating retrieval from generation, tracking evidence type and confidence, packing only relevant memories, and requiring the answer stage to identify uncertainty rather than fill gaps.

Making memories evolve

Real memories change. An object moves, a preference is corrected, and a repeated observation may reinforce an existing fact.

Building lifecycle states for canonical, superseded, stale, conflicting, deleted, and archived memories was significantly more complicated than simply appending rows to a database.

Handling natural voice interaction

Voice input includes partial transcripts, polite phrasing, short confirmations, interruptions, and ambiguous references.

Commands such as “yes,” “cancel,” “forget that,” and “actually, it is the blue one” needed conversational context and careful ordering to prevent accidental actions.

Balancing recall with privacy

A wearable memory assistant must give the wearer control.

Remember includes manual capture, private mode, confirmation before destructive actions, undo, sensitive-content thumbnail suppression, optional cloud storage, and local-only operation. Approximate location is the default, while precise location remains opt-in.

Operating within a wearable runtime

Lens Studio does not behave like a browser or a conventional Node.js application. Camera readiness, VoiceML timing, device-only APIs, texture handling, network changes, and available cryptographic primitives all required runtime-specific fallbacks.

What we are proud of

We are especially proud that Remember is more than an image-captioning demo.

It provides a coherent memory lifecycle:

1. Capture a real first-person moment.
2. Convert it into structured evidence.
3. Merge it into an evolving graph.
4. Retrieve only the most relevant memories.
5. Answer naturally without overstating the evidence.
6. Allow the wearer to correct, delete, pause, or undo.

We are also proud that the system remains useful when optional services fail. Memories can continue locally when cloud storage is unavailable, offline changes can sync later, and spoken answers fall back to the visual interface if text-to-speech fails.

Finally, the compact graph design avoids storing an unbounded stream of raw captures while preserving the relationships needed for useful recall.

What we learned

We learned that building an AI memory assistant is less about storing more information and more about deciding what deserves to survive.

Reliable recall depends on:

• Good evidence representation
• Clear uncertainty
• Strong retrieval
• Memory lifecycle management
• Conversational context
• User control

We also learned that wearable AI needs different interaction patterns from phone or desktop apps. Responses must be short, capture must feel effortless, and privacy controls must be immediately accessible by voice.

What’s next

The next stage for Remember is to make recall more personal, spatial, and proactive.

Planned improvements include:

• On-device embeddings for faster private retrieval
• Audited AES-GCM encryption when supported by the runtime
• Better multi-day and cross-session memory consolidation
• Spatial anchors that reconnect memories to physical places
• Personalized importance and forgetting policies
• Shared household memories with explicit consent
• Accessibility modes for users with memory impairments
• A companion interface for reviewing and organizing memories
• Evaluation tools for measuring retrieval accuracy and hallucination rates
• Proactive reminders when a remembered object or task becomes contextually relevant

The long-term vision is a memory layer that quietly supports everyday life while keeping the wearer in control of what is captured, retained, and forgotten.

Built With

• augmentedreality
• codex
• computervision
• gitlfs
• gpt-5.6
• lensstudio
• memorygraph
• multimodalai
• openai
• remoteservicegateway
• snapspectacles
• speechrecognition
• supabase
• texttospeech
• typescript
• voiceml
• wearablecomputing