-
-
Scrapp login, main capture page, and sticker book.
-
Spatial sticker viewing, and 3D mesh generation based on a sticker using Meshy.ai.
-
Spatial viewing of a custom collage, viewing of the collage studio, and friends page.
-
Creation and completion of a scavenger hunt using various items from Nate's desk.
-
Hunt summary page, collage editor window, and app settings.
Inspiration
We were inspired by traditional scrapbooking and the latest trend of junk journaling, and wanted to bring this beloved tradition into the digital space to make it easier to share your creations and collaborate with your friends.
What it does
Scrapp lets you turn any object in the real world into a virtual sticker. Simply put on a pair of smart glasses and say "Scrapp that." The mobile and web app allow you to view your stickers of what you saw IRL and turn them into collages. Meshy.ai integration lets you turn any sticker into a 3D model. You can add your friends to send them stickers directly in the app or export them to your photos app or favorite social media platform. Hunts let you turn the locations you scrapbook into scavenger hunts for your friends. Users can also create stickers using their phone’s camera and experience them in AR, ensuring friends without smart glasses can still participate.
How we built it
This project uses Meta Wearables Device Access Toolkit to capture images and voice commands. SAM3 is used to segment the image captured and cutout the object requested by the user. ARCore with SceneView is used to place stickers and 3D models in the real world. A three.js page allows users to create a 3D collage. The backend is handled with MongoDB Atlas. For the scavenger hunt feature, OpenCV SIFT feature matching checks a photo against the hunt's targets. The check-off only counts within a set distance of the target, and progress is shared between friends through a leaderboard. Meshy.ai allows users to turn their stickers into 3D models. Segmentation models were run locally and accessed through a Cloudflare tunnel. The mobile app was created with Flutter.
Challenges we ran into
Most of the challenges we ran into stemmed from the resource heavy segmentation models bogging down our hardware. It was also a challenge refining the segmentation model to give clean stickers every time and getting accurate voice commands without false triggers.
Accomplishments that we're proud of
We are proud of how polished we were able to make this project in the time we had. Scrapp really feels like something that could be a fun part of people's daily lives.
What we learned
We learned a lot working within the constraints of screen-less smart glasses. As this was a first for all of us working with the Meta Adventurers, we were challenged to come up with ideas that took advantage of the hardware and provided a unique user experience without visuals on the wearable. Ultimately, this constraint became one of the application's strengths, allowing its core features to remain accessible to everyone while offering meaningful usability benefits, such as hands-free capture and seamless interaction, for those using smart glasses.
What's next for Scrapp
More social features, more viewing options for scrapbooks
Built With
- dart
- flutter
- html
- javascript
- meshy
- python