Inspiration

Relay started with a familiar question: where did I leave my keys? Building an answer showed us that the useful part was never the arrow pointing at a hook. It was the memory behind it: knowing what was seen, where, when, and what may have happened since.

People don't watch their homes continuously. We remember what we saw, assume most of it still holds, notice when something may have changed, and revise when we look again. A kitchen makes this obvious. You "know" there is flour in the corner cabinet, roughly how much, and that you used some on Sunday. You also know you should check before promising pancakes.

So Relay became an attempt to give AI that kind of memory: an evidence-based, revisable understanding of a person's physical world, used to help at the moment it matters and then get out of the way.

What Relay is intended to do

Relay is designed as a personal environmental intelligence. It builds and updates an understanding of physical objects, their locations, quantities, events, dependencies and the person's intentions, then uses that understanding to give timely, minimal assistance.

The intended loop is observe → remember → reason → assist → observe the outcome → revise. Relay should keep the best-supported current understanding, keep a record of how it got there, and know which observation it needs next. We call this shared, revisable model the Environmental World Model: one memory of objects, places, quantities and events that every Relay application reads from and every credible observation updates.

It is meant to be broader than three things it can be mistaken for:

  • A lost-object finder answers one question about one object. Relay's memory is designed to be shared across everything it has seen, so that one record can answer "where are my keys?" and "can I make dinner?". Today the keys and kitchen parts still use separate memory modules.
  • A pantry database needs someone to maintain it. Relay is meant to learn from ordinary activity: opening the fridge, turning a cabinet, putting groceries away. It should say plainly when its knowledge is old or second-hand.
  • A conventional AR interface puts panels in front of you. Relay is meant to stay out of sight until a question or task makes something relevant, then place a small cue at the physical location and remove it when the job is done. Opening a real door or picking up a real item moves the task forward, not a virtual button.

Glasses are one interface, not the system. The same memory could later take observations from a phone camera, fixed household sensors or a robot, with permission.

The proposed architecture (proposed, not built as a whole):

  1. Observation adapters for glasses, phones, recorded video and other sensors. Each says what kind of evidence it provides.
  2. Evidence admission and identity association. Inputs are checked, provenance is kept, observations are linked to particular objects, and ambiguity is held open rather than guessed.
  3. An environmental world model. Objects, containers, rooms, events, tasks and reference frames are linked by explicit relationships, for example object located_in container, event changes_quantity object, task requires object, object may_substitute_for object.
  4. Belief and intention reasoning. It estimates what is probably true now, keeps quantities consistent with new evidence, works out what a task depends on, and decides what is worth checking.
  5. Applications that share this one model.
  6. Feedback and correction. Outcomes and user corrections revise beliefs, without blurring evidence and inference.

The first two applications are Spatial Recall ("Where did I leave my keys?") and Environmental Inventory ("What can I make for dinner, and what should I check first?").

The full kitchen experience we are aiming for (proposed, not built):

  • You ask by voice what you can make with what you have. Relay asks a useful follow-up ("What are you in the mood for?"), suggests meals, and asks you to check an ingredient when it is unsure.
  • It recognizes ingredients and estimates their quantities from the glasses' camera.
  • It keeps its evidence across sessions, rooms and days, and reconciles it with what you use and buy.
  • The same memory serves other intentions, such as finding your keys while you cook, without losing what it knows.
  • It should feel as responsive and spatially continuous as a well-made game, with no scores, effects or distractions.

What the current software demonstrably does

We keep six categories apart, because the vision is much larger than what exists today.

1. Implemented and independently reviewed. An independent verification agent froze its tests before reading each candidate and then checked the candidate by hash.

  • A kitchen memory engine (one plain TypeScript file; independent review confirmed that the final Lens holds a byte-identical copy of the reviewed version). Working on simulated observations, it does the following:
    • It keeps the last credible sighting of each ingredient, with its place and time, and keeps the kind of evidence behind it: simulated sensing, or something the user said.
    • Amounts are kept as ranges, never a single false-precise number.
    • "Not seen" does not mean "gone". Only a full, tracked look at a place can say an item is no longer where it was, and even then its new location is unknown.
    • It says "You can make pancakes" only when every ingredient is in view now. Otherwise it says "You can probably make pancakes; some of it may have changed since it was last seen."
    • An item known to be gone or moved is never shown as a current amount, never counted toward a recipe, and never guided to.
    • It answers "what can I make for dinner?" from that memory. It names what to fetch, what to check first and what is missing, and it offers substitutions.
    • On a rotating cabinet shelf it guides to the shelf rather than to an old point the shelf may have turned away from.
    • A new dinner question or a newly chosen meal starts its guidance fresh, so it never reports "everything has been found" from an earlier round.
    • It produces no confidence percentages.
    • All of this passed independent review at the engine level.
  • The Spatial Recall memory logic (last-seen place and time, "I haven't seen them since", no "in view" claim after the keys leave view) passed independent review in an earlier frozen build. That review covered its memory tests and its scripted Preview run, checked from logs. Its copy inside the combined Lens was not part of the kitchen end-to-end review.

2. Implemented and independently reviewed in Lens Studio Preview, with one quality check still open.

  • The integrated kitchen Lens in Lens Studio Preview. As built:
    • Storage: a refrigerator, a freezer and a pantry cabinet, each with a door you drag open, and a corner cabinet with a rotating shelf.
    • Ingredients: eight, shown as objects: eggs, milk, butter, ground beef, flour, sugar, pasta and canned tomatoes.
    • Sensing: an ingredient is "seen" only when it is in view, in range and not blocked. A small "Simulated sensing" label is always visible.
    • What it remembers, on screen: at a storage place in view with its door open, the Lens shows the engine's own line for each remembered item there. Under each dinner answer it shows up to three of the engine's reasons.
    • Dinner: the engine answers the dinner question for pancakes or pasta bolognese, with a meatless substitution.
    • Guidance: a small arrow at the edge of view and a marker at the remembered place, on the shelf for the corner cabinet.
    • On-screen controls: the dinner question, the meal choice, "Report: I bought eggs" and Stop.
  • What independent review of the final Lens confirmed, in Lens Studio Preview with simulated sensing, for each of the seven behaviours we set for the kitchen:
    • Observed at several places. Four storage places appear as 3D cabinets holding ingredient objects, and the corner shelf's cans turn with the shelf. An ingredient is recorded only when it is in view and not hidden behind a closed door.
    • Retained after leaving view. After you look away, every item is still held as last seen at its place, never as in view.
    • Quantities change with evidence. A simulated use of flour lowers it on screen from about 1,000 g to 800 g. "I bought eggs" is kept as what you said, never as seen. When a look finds the eggs gone from their spot, no amount or place is shown for them.
    • One-tap request. The dinner question, asked with one button, is answered at once.
    • Honest recommendations. The answer, the meal choices and the reasons on screen are the engine's own text, with "probably" where it is unsure. When the flour runs out, the answer changes to pasta bolognese, and the pantry line shows the flour as used up.
    • Guidance to the real place. Guidance points to the remembered place of each ingredient, including the shelf after it turns, and to the shelf rather than a precise spot when the exact spot may have changed.
    • Updates come from evidence. Each step completes only when its item is seen. No control the wearer uses changes memory except the labelled "I bought eggs" report; the test-mode buttons that simulate household events are labelled and off by default.
    • A new dinner question no longer shows "has been found" from an earlier round.
  • Guidance while you turn was then checked in motion in the recorded run.
  • What is still open. The review did not pass the candidate overall, because one quality check is open:
    • The Preview stall (below) is not yet isolated or bounded. We are entering with this reliability limitation open, and the investigation continues after the competition.
  • Our own scripted run. In our final scripted run of the frozen candidate, all 29 test scenarios passed. That is the builder's count, not independently reproduced.
  • Known defect: Preview intermittently froze while running our Lens and tests. In our final candidate's runs, pauses of up to about 4 seconds appeared, and the run still passed. The pauses coincided with the host running short of memory, which may be one trigger; that is not yet shown. We have not run a deliberate bounded series, so the stall is not isolated, not bounded and not fixed. Independent review rates it open: the required series of repeated runs has not been done.

3. Simulated sensing. Every observation in this Lens is simulated in Lens Studio Preview. The scene decides what counts as a sighting from visibility, and simulated amount readings stand in for measurement. No camera image has been processed and nothing is recognized. Every Preview shot in the video is labelled this way.

4. Real recognition experiments (separate from the Lens).

  • What exists. We built a recognition pipeline that runs locally, with no cloud service, on recorded phone video:
    • printed markers on each surface let it place each detected object on that surface;
    • it matches objects that were enrolled beforehand;
    • it holds look-alike items as unresolved candidates instead of guessing;
    • it never concludes that something is gone from a missed detection.
  • Independent review. Our reviewer accepted the pipeline for the real experiment, with conditions, after tests on synthetic images. On those synthetic test runs it produced no false-certainty failures, but it was not yet useful enough to pass our own go threshold.
  • Real footage. No real household footage has been processed yet. The go/no-go protocol for the ten-object experiment was frozen before any real run.
  • Hardware. A phone result would not be evidence about Spectacles.

5. Proposed, not built. The environmental world model as a whole, relationships as general first-class data, cross-room memory, persistent identity, the twelve application areas below, and the experiments in "What's next" beyond the pipeline in category 4.

6. Untested on hardware. Relay has not run on Spectacles or any glasses. We have not yet exercised SnapML, the Depth Module or Depth Cache, World Query, Spatial Anchors, Custom Locations or the Navigation Kit. They are planned for evaluation.

How we built it

  • Planning. We planned the Lens with the Devpost Learn skill pack's plan-lens interview in a coding agent:
    • the first session planned the Spatial Recall Lens;
    • a second, kitchen-specific plan-lens session planned the kitchen scope. Its brief keeps what the current Lens does apart from the full experience we are aiming for.

The kitchen brief is in the repo's devpost/ folder.

  • Building. We built the Lens in Lens Studio through CLAD and the Lens Studio MCP, driven by Claude Code on a Mac. The agent built the scene and the observation, guidance and UI scripts through the MCP:
    • the simulated observation step: a frustum check, a range check and a physics ray that must reach the item first;
    • doors and the rotating shelf moved with Spectacles Interaction Kit hand drags;
    • the guidance arrow and marker;
    • controls only for genuine software decisions (ask, choose a meal, stop), plus one labelled user report ("I bought eggs"), which memory keeps as "you said";
    • labelled test controls for simulated events, such as "someone used 200 g flour", which appear only in a test mode. Our scripted run switches that mode on only around those labelled events, and independent review confirmed every test-control press fell inside those windows.
  • Testing. We wrote LEAF scenarios that drive the simulated hand and camera in Preview and check behavior, not just compilation.
  • Engine. The kitchen engine was written as a separate, dependency-free TypeScript file by another AI coding agent session. It was tested in Node, including long-run and mutation tests, and reviewed independently before CLAD integrated it. Independent review confirmed that the final Lens holds a byte-identical copy of the reviewed engine.
  • Verification. A separate verification agent wrote acceptance tests from the requirements before reading any code, and reported PASS, FAIL or NOT VERIFIED for each.

Challenges and failures

  • The Preview stall. Preview intermittently froze while running our Lens and tests. In one session, two of three full test sequences stopped mid-scenario with Lens Studio near 200% CPU, and only a Preview refresh recovered them. We first suspected the engine and fixed a real per-frame growth problem: the engine had been keeping a little more history every frame. The stall continued in that session. In two later sessions it did not recur in more than 50 runs, which also means we could not isolate it. In our final runs it reappeared as pauses of up to about 4 seconds (see Known defect). The cause is not established, and it is not fixed. Until it is, we do not describe the experience as reliable.
  • Stale certainty. One of our own probes found that an earlier engine version said "You can make pancakes" from looks up to six hours old. We changed it so that it says so plainly only for what is in view now, and "probably" otherwise. That change passed independent review.
  • False "done" messages. Independent review found two cases where the guidance could say "everything has been found" while an ingredient was still missing or unchecked. One was the user choosing a meal that couldn't be made. The other was an "I bought eggs" report with nowhere recorded to look. Both were fixed by making the closing line name what is still missing or still to check. Independent review of our integrated Lens then found a third: after a new dinner question, the screen could still say "Everything for pasta bolognese has been found" from an earlier round. We changed the engine so each new request starts its guidance fresh, and that fix passed independent review at the engine level. Independent review then confirmed it in the integrated Lens for a new dinner question.
  • Showing enough. Independent review of an earlier candidate found that the engine's amounts, last-seen times and reasons never reached the screen; the wearer saw only the answer and the guidance line. The memory was right, but the wearer could not check it. We then put the engine's own lines and reasons on screen, at the storage place and under the answer, and independent review confirmed them there.
  • Preview limits. In Lens Studio Preview the Spectacles camera API returns no cameras, so every observation in this Lens is simulated. Our Lens also keeps its memory for one session only.
  • Tooling friction.
    • A test-framework build for a newer Lens Studio broke compilation, so we pinned an older LEAF.
    • A capture tool auto-installed packages that broke the build.
    • Preview delivers about 20 drag updates a second, which made our door-opening tests fail until we adjusted them.
    • Once, a scene change was not saved, and independent review caught that the "new" candidate was identical to the old one.
  • No hardware. We had no Spectacles available, so nothing here is evidence of on-device performance.

Verified accomplishments

  • A kitchen memory engine that passed independent review at the engine level. That review included six specific checks that a gone or moved item is never presented as available, counted or guided to.
  • A memory that keeps a user's report ("I bought flour") separate from a sighting. It is labelled "you said". It can raise what might be there, but it never raises what we know is there, and it never becomes a fake camera observation.
  • A Spatial Recall memory whose logic passed independent review in an earlier frozen build (memory tests plus a scripted Preview run checked from logs).
  • A kitchen Lens in which independent review passed each of the seven behaviours we set: observed at several places, retained after leaving view, quantities changing with evidence, a one-tap request, honest recommendations, guidance to the real place, and updates only from evidence. It is shown in Lens Studio Preview with simulated sensing; one quality check, the Preview stall, remains open.

What we learned

  • Honesty has to live in the data model, not the copy. If "seen", "remembered", "reported" and "unknown" are separate states, the interface can't accidentally overclaim.
  • "Not seen" is not "gone". Most false certainty comes from treating a missed glance as evidence of absence.
  • Position belongs to containers. Flour on a rotating shelf moves through the room while staying put on the shelf. When the shelf's orientation is unknown, the honest cue is "the shelf", not a precise point.
  • Independent review finds what self-testing misses. Both false "done" messages and an unsaved scene were caught by independent review. For the engine, the reviewer wrote its tests before reading the code.
  • Simulation is useful but bounded. It let us build and test the reasoning and interaction layer without hardware, and it proves nothing about recognition.

The six hard problems still ahead

Each line gives the problem, our proposed approach, and where we are today.

  1. Reliable object recognition. Proposed: combine visual detection with label reading, packaging, geometry and scene context. Use local models first. Treat recognizing a category ("a flour bag") as different from recognizing your item. Never let a confident guess enter memory on its own. Depth sensing helps with geometry, but it doesn't tell you which object you're looking at. Today: the Lens is simulated. A separate recorded-video pipeline matches enrolled objects and has been tested on synthetic images only (category 4).
  2. Persistent identity over time. Proposed: link sightings by appearance, position, relationships and continuity. Never assume that two identical-looking items are one, and leave ambiguous matches unresolved. Today: the recorded-video pipeline holds look-alike items as unresolved candidates, tested on synthetic images only. Identity over days and across rooms is not built.
  3. Cross-room localization. Proposed: device tracking, persistent anchors, landmarks and a graph of connected rooms. Withhold or qualify guidance when the reference frame is in doubt. Spatial Anchors can restore a position, but they don't save what we know about it, so that knowledge has to be stored by us and re-attached. For homes we would avoid mapping features that upload room scans. Today: single scene, single session. The memory already ignores sightings made while tracking is lost.
  4. Durable coordinates in movable containers. Proposed: store positions relative to shelves, drawers, fridges and bags, and never invent a precise spot when a container's orientation is unknown. Today: the engine guides to the rotating shelf as a whole, at place level.
  5. Calibrated uncertainty. Proposed: conservative evidence states and quantity ranges first. Introduce numeric confidence only once it can be measured against ground truth. Today: evidence states and ranges are implemented, with no numeric confidence.
  6. Trustworthy evidence. Proposed: a typed admission boundary that keeps the source, time, frame, sensing status and uncertainty of every accepted observation. Keep history separate from current belief. Support deletion, and test forged inputs. Storage integrity does not prove that an observation was true. Today: in the engine, answers name the kind of evidence they rest on, and only observations and labelled reports change memory. In the final Lens, independent review found that no control the wearer uses changes memory except the labelled "I bought eggs" report; the test-mode buttons that simulate household events are labelled and off by default.

Where the same intelligence could go

These are possible extensions of one shared memory, not features we have built: physical task continuity (returning to an unfinished repair); preparing and packing for outings; predictive household maintenance; resuming interrupted goals; planning materials for projects and repairs; situated navigation and discovery; emergency preparedness and supply awareness; managing belongings such as clothing, tools and supplies; noticing change in plants, equipment and rooms; procedural help at work; cognitive accessibility and independent living; and a shared, permissioned environmental memory for future household robots and other agents.

The point is that these should not be twelve separate products. They are twelve uses of one memory.

What's next

The next critical step is real sensing, measured against physical ground truth. More simulation won't get us there.

  1. Ten-object experiment. Ten real household objects, including look-alike packages and partly used ingredients, in a controlled space. Someone other than the system records the ground truth before the run and withholds it from the system. The sequence:
    • observe the objects;
    • record their containers;
    • leave;
    • recall them;
    • move some objects;
    • rotate a container;
    • report additions or use without showing them;
    • look again;
    • answer.

The primary failure measure is false certainty: any claim that something is available, or at a precise spot, that the ground truth contradicts. We will also count unknown-object rejections, false merges and identity switches. Thresholds are fixed before the first run. The first sensor may be a phone camera, and that result would not be evidence about Spectacles.

  1. Two-room localization experiment. Covers walking between rooms, losing tracking and restarting.
  2. Quantity experiment. Real consumption measured against our ranges.
  3. Platform evaluation. We will evaluate SnapML, Depth Cache, World Query and Spatial Anchors on Spectacles when hardware is available. A future Spectacles version would run recognition on the device itself (Camera Module and SnapML), with no imagery leaving the glasses. This is not built or tested. Some depth and semantic-query features are documented as Experimental, and Lenses that use Experimental features cannot be published to a wider audience, so they are research tools for us, not product dependencies. We will also bring the same memory to Meta Quest through a platform adapter rather than a second memory system.

Disclosure

  • All sensing shown in the Lens is simulated in Lens Studio Preview.
  • The Lens was built during the submission period. Our source is released under the MIT License. The Lens Studio packages the project uses (Spectacles Interaction Kit, Spectacles UI Kit and LEAF) are not included in the public repository. They remain under Snap's own terms and are not covered by our license.
  • Kitchen engine. It was written by a separate AI coding agent session and integrated by CLAD. It is a simplified public version of a more complete private memory design by the same team. No private code is included.
  • Related work. The submission period began October 15. Some of our related work predates that date. The related work we have records of:
    • our broader agent-governance software, which predates the submission period;
    • a separate Meta Quest spatial-interface prototype, first committed on September 28, 2026, which also predates it;
    • a preparatory workspace for this entry, with preliminary requirements and sensing investigations. Our first record of it is a status report dated November 3, 2026.

None of this work contributed code to this project. The Lens project itself was created after Lens Studio was installed on November 4, 2026.

Built With

Share this project:

Updates

Submission history