Reflection

Operational memory for the physical world

Comment

Inspiration

Enterprise knowledge systems index documents, tickets, chat and code, but a large share of operational work happens away from a screen: on factory floors, in hospitals, warehouses, substations and data centers. Observations from that work (equipment state, faults, repairs, context) are rarely captured in a structured form. They persist in individual memory until a shift ends and are then transferred verbally or lost. Reflection addresses this gap by capturing physical work as it happens and turning it into a structured, queryable record in which every entry is backed by evidence.

For a data center technician: a PSU fault observed at 2 a.m. and the corrective action taken are recorded, attributed and available to the incoming shift, rather than left on a sticky note.

What it does

Field personnel wear smart glasses and capture a photo at each meaningful event. Reflection converts these captures into an event graph that both people and AI agents can query.

  • Low-friction capture: Each photo is stored with wearer attribution, timestamp, location and an optional spoken note. Faces are blurred on-device before upload.
  • Asset resolution: Text read in the scene (labels, tags, panel identifiers) is matched against the organisation's asset registry, so each event is linked to canonical assets.
  • State tracking: Each observation is compared only with the most recent prior observation of the same asset. State transitions are recorded as change edges, and fault conditions raise alerts.
  • Closed-loop resolution: A later observation showing an asset back in a healthy state automatically resolves the open alert and work order, citing that photo as evidence.
  • Shift handoff: An auto-generated report lists open alerts and work orders, state changes, area coverage by wearer, and field notes, with every line citing its source event.
  • Agent access: An MCP interface exposes the record to AI assistants. Answers to natural-language questions are grounded in, and cite, specific events.
  • Spatial view: A 3D site map renders wearer tracks, capture locations and open issues, with time-based replay.

For a data center technician: a photo of a PDU showing "PSU2 FAULT" opens an alert on that asset; a follow-up photo after the swap closes it. The incoming technician's handoff reflects both, and a supervisor can query the status remotely.

How we built it

Reflection is composed of four layers:

Capture (iOS + Meta Ray-Ban glasses)

  • A Swift app built on Meta's Wearables Device Access Toolkit. Hardware shutter events trigger capture, including while the app is in the background.
  • On-device face blurring, speech-to-text for field notes, and GPS with barometric floor detection.
  • A persistent upload queue with retries keeps captures intact under intermittent indoor connectivity.

Memory (SpacetimeDB)

  • A single SpacetimeDB module on Maincloud holds all state (events, assets, graph edges, alerts, work orders, episodes, handoffs); there is no separate application server.
  • Procedures inside the module run ingestion, perception, episode segmentation, alert resolution, handoff generation and an agent tool loop, including the outbound calls to the models.
  • Clients receive changes through real-time subscriptions, so a photo reaches the console within seconds.

Perception and reasoning (NVIDIA Nemotron)

  • Nemotron 3.5 Super VL extracts structured data from each image: scene, objects, text in scene, state changes and anomalies.
  • Nemotron 3.5 Lightning powers the tool-calling agent, episode titling and handoff summaries.
  • Azure OpenAI (gpt-4o, gpt-5) serves as an automatic fallback, and Azure AI Vision provides image embeddings for similarity search.
  • An MCP server exposes 11 tools to Claude Desktop and other agents, returning photos alongside citations.

Console

  • A web console with a Three.js 3D site map, a shift timeline and work-order tracking, deployed on Vercel.
  • A synthetic data pipeline generates photoreal sample shifts for five verticals (data center, manufacturing, hospital, utility, warehouse), with each image validated by the vision model before use.

Challenges we ran into

  • Grounding generated output: An early version of the handoff, written entirely by the model, included unsupported statements. We restructured it so every factual section is built deterministically from the database, and the model writes only a summary in which each sentence must cite a valid event or is discarded.
  • Entity attribution: A single frame often contains several assets (a rack, the equipment mounted in it, cable tags). We built resolution rules that attribute each finding to the correct asset and prefer specific equipment over its enclosure.
  • Temporal comparison: Change detection has to compare an asset with its own previous state, not with whatever photo came immediately before. We give the model asset-tagged history and accept a change only when there is a prior observation of that asset.
  • Hardware constraints: Full-resolution capture takes roughly 40 seconds to transfer over Bluetooth, and indoor positioning is accurate only to tens of meters. We measured both on real devices and tuned capture settings accordingly.

For a data center technician: a close-up shows the rack, the PDU and a cable tag. The fault must be attributed to the PDU, not the rack.

Accomplishments that we're proud of

  • About 5 seconds from shutter to an indexed, asset-linked event on real hardware.
  • A complete detect, track and resolve cycle driven entirely by captured evidence, with no manual data entry.
  • Full provenance: every handoff line and every agent answer resolves to a photo, a timestamp and a wearer.
  • The entire backend, including the agent loop, runs inside a single SpacetimeDB module.
  • One architecture validated across five operational domains.

What we learned

  • Capture has to fit the workflow: field personnel won't type, but they will press a button and speak a sentence.
  • Models are effective at summarisation but unreliable at maintaining factual lists; structured facts belong in the database.
  • Correctly identifying what an observation refers to matters more than raw perception quality. A well-maintained asset registry adds more than a larger model.
  • Privacy controls (on-device face blurring, counting people but never identifying them) are a prerequisite for workplace adoption.

What's next for Reflection

  • System integration: Bidirectional sync with CMMS, ITSM and DCIM platforms: create work orders from alerts and import asset registries.
  • Guided workflows: Agent-issued field tasks, completed and verified by photo.
  • Additional capture sources: Mobile devices, body-worn cameras, fixed cameras and robotic platforms feeding the same pipeline.
  • Multi-site deployment: A federated memory across facilities, with per-site access control and retention policies.

For a data center technician: a supervisor dispatches "inspect PDU-B" to the technician's glasses; the verification photo closes the corresponding ticket in the ITSM system automatically.

Built With

Share this project:

Updates

Submission history