SnapSEEK

• 

  Hero Image

• 

  Capture

Inspiration

We were inspired by the idea of turning the everyday world into an interactive, story-driven space—something playful yet meaningful. With the rise of wearable AR like Snapchat Spectacles and the power of visual AI, we saw a unique opportunity to reimagine how people engage with their surroundings using just a pinch gesture and a bit of curiosity.

What it does

SnapSEEK is a gesture-powered AR experience for Spectacles that lets users "frame" parts of the real world with their fingers, crop the scene, and send it to an AI system for recognition. From there, the app unlocks interactive narratives, object-based challenges, or playful experiences. Think: scavenger hunts, avatar appearances, or real-world object spotting games—all activated by natural hand gestures and powered by LLM-based visual understanding.

How we built it

We developed SnapSEEK using Snapchat’s Lens Studio with Spectacles support, leveraging hand-gesture recognition—specifically the index finger and thumb pinch—for intuitive input. When a user frames part of the scene, the captured image is sent to a backend powered by OpenAI’s GPT for visual scene analysis. ChatGPT then returns a caption and identifies relevant keywords to drive the next interaction. Our prototype builds on Spectacles' example code and integrates Lens Studio tools such as Tween Manager, Shader Graph, and Behavior Scripts. All in-scene 3D assets were generated using GenAI based on custom prompts, allowing us to quickly visualize playful and responsive AR content.

Challenges we ran into

• Integrating gesture recognition in a consistent and reliable way across lighting conditions
• Latency and data handling when sending cropped frames from AR to a cloud-based AI service
• Designing intuitive UX for a hands-free, heads-up interface
• Balancing technical feasibility with the playfulness we envisioned

Accomplishments that we're proud of

• Built a fully functional gesture-based image capture system for AR glasses
• Transformed cutting-edge tech into a playful, shareable experience
• Successfully integrated LLM-powered scene understanding to drive responsive AR interactions
• Delivered an experience that feels intuitive, delightful, and socially engaging

What we learned

We learned how powerful and expressive hand gestures can be when designed thoughtfully. We also gained insight into how users respond emotionally to AI-driven AR experiences—especially when the system reacts with surprise, humor, or delight. On the technical side, we sharpened our skills in Lens Studio scripting, LLM visual parsing, and realtime user interaction design.

What's next for SnapSEEK

• Adding multi-player support for competitive object-finding
• Creating localized versions of hunts with cultural narratives
• Allowing users design their own creative re-framing theme and build challenge community.

Built With

• chatgpt
• figma
• lensstudio