Inspiration

Every building has a story. Most people will walk past it every day and never know.

Take Stanford's Engineering Quad. Thousands of students, visitors and staff cross it every day, past buildings named for the people behind NVIDIA, Microsoft and HP: Huang, Gates, and Packard. Behind each name is a story about how modern technology was built. But nothing on the walk tells you. At most there's a plaque, and when was the last time you actually read a plaque?

One of us grew up in New York and LA. My favorite moments from my childhood were walking through museums and learning more about the world. Even outside of museums, the streets and buildings are just as full of history, and they have so much to say. Retrospecs is the museum I wished I could take outside.

We built Retrospecs so the places you pass every day can finally tell you their stories.

What it does

Retrospecs is an AI tour guide for Snap Spectacles that turns the buildings you walk past into stories you can hear.

  1. Arrive: a 3D map of the Engineering Quad unfolds in front of you, with a pin showing where you're standing.
  2. Ask: look at any building and pinch, or ask a question out loud.
  3. See: while you stand outside in the Engineering Quad, you will see the names of each building inviting you to learn more. Pinch one, and the guide tells you its story. Buildings you've walked past a hundred times suddenly have something to say.
  4. Explore: the building grows into focus while the rest of campus fades back, with illustrations and a small 3D animation of its story. At Huang, a chip fires its cores one at a time, then all at once: the idea behind NVIDIA's GPUs, told right where it's named.

Every answer is grounded: the guide can only speak from 32 landmarks and 86 facts, each with a source.

How we built it

  • Lens Studio and TypeScript on Snap Spectacles, built on Snap's MIT-licensed Depth Cache sample.
  • A generated 3D map. A script pulls building footprints, heights, roads and bike paths from OpenStreetMap through the Overpass API and bakes them into data the Lens loads directly: 562 buildings within 1.1 km of Huang. At runtime the footprints become meshes, split into chunks to stay under Spectacles' vertex limits.
  • Grounded AI answers. Gemini, through Snap's Remote Service Gateway, answers voice questions, but it must choose from our fact cards. A validator drops any building, fact or number that isn't on a sourced card. Text-to-speech speaks the answers.
  • Focus mode and nine 3D animations, built procedurally in code so they animate without rebuilding meshes.
  • Calibration: one pinch at a known starting point lines the map up with the real campus, with a two-pinch wall method as a fallback.
  • The core logic is unit-tested.

Challenges we ran into

  • Pivots. We first tried generating historical video of campus with an AI world model, but couldn't make it historically accurate. We moved to a design where every fact is real and sourced.
  • Keeping an AI guide honest. A building's story is only worth telling if it's true. Language models make things up, so we built fact cards and a validator to keep every answer tied to a source.
  • Lining up a virtual map with the real world, and making calibration work both outdoors and at a judging table.
  • Designing for an additive display, where dark colors become transparent. We built a quiet sandstone map so the one selected building stands out.
  • Missing data: OpenStreetMap has no height for many campus buildings, so we had to keep the map readable anyway.

Accomplishments that we're proud of

  • A full 3D map of the Engineering Quad generated from open data, so any place in the world can get one.
  • 86 sourced facts about buildings most people never think twice about, and an AI guide that says "I don't know" instead of guessing.
  • Animations that tell a building's story where it happened, like the GPU chip lighting up beside Huang.

What we learned

  • The most interesting stories aren't in famous landmarks. They're in the ordinary buildings people pass every day without a second look.
  • People don't dislike history. They've just met it as dates in a textbook or a sign nobody reads. Standing in the place, and hearing its story there, changes that.
  • Grounding an AI in sourced facts matters more than making it sound clever.

What's next for Retrospecs

  • More places: the rest of campus, then other campuses, neighborhoods and cities, starting from OpenStreetMap. Every street has buildings with untold stories.
  • Community stories: a way for locals to add their own sourced stories, not only official ones, because some of the best stories are the ones only neighbors know.
  • Point-to-guess: a game that's already built, coming back once it's polished.
  • Real-world testing with everyday walkers, campus visitors and tour groups: do people start noticing the buildings around them, and remember their stories a week later?

Team: Jason Anderson · Sayra Kurtoglu · Angela Liu

Credits: Map data © OpenStreetMap contributors (ODbL). Built on Snap's Depth Cache sample (MIT).

Built With

Share this project:

Updates

Submission history