Matterly

Inspiration

We wanted to design an experience that fits for the purpose of the Spectacles and the mission of Snap, part of which includes empowering people to live in the moment, learn about the world, and enhance our relationship with it. We asked ourselves “What topics could we educate with Spatial AI that still goes unnoticed everywhere else?

Our answer was to use the Spectacles to increase people’s knowledge and understanding of the makeup of everyday objects. Helping people become more mindful of the ecosystem lifecycle of our surroundings and encouraging better choices that extend the health of our planet. We also felt conventional sustainability teaching falls short. It over-emphasizes what happens after people buy something, like which bin and how to sort, while overlook the choices that matter most: buy less, buy durable, reuse, repair.

What drew us to Spatial AI is that it closes a gap textbook learning never could. Abstract facts stay dissociated from the objects they describe. By teaching in-situ, with knowledge bound to the real object, the lesson stops being something people read about and becomes something they experience in the moment.

What it does

Matterly scans the objects around you to recognize what they are and what they're made of, whether you're holding something in your hand or looking across a room, no barcode or lookup needed. Reveals the object's hidden lifecycle, anchored to the real object in the space: where it came from (extraction, processing, manufacturing) and where it really goes (use, disposal), , staying honest about the fate most objects actually meet.
Shows the real material cost in carbon and water, pulled from a citable materials database and research papers, not made up by the AI.
Answers your questions through a voice AI sustainability advisor you can ask anytime.
Compares multiple objects at the moment of purchase, weighing them across their full lifecycle, so you can make a better choice on the spot.
Collects what you learn in a companion web portal, an object-dex where each scanned object unlocks its story, impact, and fate.

How we built it

We started with base learning block examples from Snap to get foundational tools in place such as DepthCache and AI Playground. We assembled what we needed from those and split the work across the team: UI/UX and interactions, AI pipelines and DB integration, and backend sync with our companion webportal.
For the AI layer, we use Gemini 3.5 Flash to detect objects and their materials, and Gemini Live to power the voice AI Q&A with vision intelligence.
A deliberate design choice was our database-first system with AI as fallback. Lifecycle stages and impact figures live in a prestored, citable database; the AI handles recognition and fills a structured form when an object isn't already covered. This keeps the system flexible enough to handle many objects while staying accurate, fast, and reliable, and it saves energy and cost on every repeated scan.

Challenges we ran into

While the vertical field of view allows for a more natural pillar area of content visibility, it limited what we could do with our hands horizontally so we had to constrain the interaction area accordingly.
When using the automatic speech recognition (ASR) module, race conditions would at times block the listening functionality, so this had to be deconstructed to isolate the issue.
When using Lens Studio AI, it would sometimes overwrite files and use historical AI memory from previous sessions in other projects, injecting scripts and Scene Objects based on those old prompts.

Accomplishments that we're proud of

The core concept itself. By making an object's real material cost visible, the extraction, water, and energy behind something used for minutes, Matterly builds the intuition that things are not easy to come by, which is what actually shifts behavior toward conscious consumption. And building it made us more materially literate ourselves: we genuinely learned the lifecycle stories we set out to teach.
A counterintuitive database-first system with AI as fallback. Most teams would have AI generate everything on every query; we prestore lifecycle and impact data and only call the AI for recognition and unlogged objects. More accurate, more reliable, and far cheaper in energy and cost across testing and repeated use.
How we designed our voice AI sustainability advisor. Three rules: no preaching, keep it relatable, and keep it fun by breaking common misunderstandings and myths.

What we learned

Building on top of Snap's sample building blocks is a very efficient way of prototyping. DepthCache and AI Playground got us to a working foundation quickly.
AI doesn't always need to go first. Our database-first approach taught us that the AI is best used where it's uniquely strong (recognition), while prestored, citable data handles everything it's unreliable at. Less energy, more accuracy.

What's next for Matterly

Location-aware disposal: the correct way to dispose, specific to where you are (rules differ by city and bin), fed into both the AR visual reveal and the AI advisor.
New material domains: food and textiles. Food brings compost and waste stories; textiles bring fast fashion, polyester, and the reuse-over-recycle lesson.
Retention system: badges, milestones, and seasonal challenges, every one earned through scanning on Spectacles, so the portal keeps encouraging to explore material stories.
Classroom mode: Students scan objects on Spectacles and engage with material stories, and everyone's discoveries pool into a shared collection, turning it into a group learning activity.