LockedIn - Vision Pro

Immerse yourself in LockedIn’s focused environment with ambient sound, explore 3D visualizations from your textbooks, and enhance your learning experience through interactive and dynamic exploration.

Comment

Inspiration

Thanks to the generous support of vGIS Inc., we had the incredible opportunity to work with a physical Apple Vision Pro at Hack the North! We developed a VisionOS app that transforms reading into an interactive and engaging experience, making textbooks come to life!

What it does

LockedIn goes beyond a standard PDF viewer by leveraging AR to render pages as immersive 3D objects. It brings diagrams to life through interactive models, allowing you to engage deeply with the material.

Our core feature “Lock in” gives a focused, distraction-free reading environment that enhances your experience—whether it’s a challenging research paper or your favorite book.

If there’s an equation, you can graph it in your interactive AR space. Reading about chemistry? Turn your reactions to life with 3D models of molecules. Love physics? Space out about it in a virtual observatory. It’s time to lock in.

The app also simplifies content with page summaries, making complex information easy to grasp. Plus, there's an interactive voice-activated chatbot that answers any contextual questions you have along the way, based on the page you’re reading.

And for added convenience, it includes a Shopify store where you can purchase and add books directly within the app.

How we built it

To bring LockedIn to life on VisionOS, we utilized a combination of SwiftUI and UIKit in the Swift programming language for display and AR/VR interactions. 3D models were created using Reality Converter and rendered into immersive environments with RealityKit.

To optimize the immersive space, we developed a custom LSTM machine learning classification model using Python, PyTorch, NumPy, and Pandas to determine a suitable environmental setting.

For the database, AWS S3 handled book files, while Convex managed metadata. We also implemented a seamless file transfer pipeline using MASV. Our backend was powered by Flask, serving as the REST API, initially deployed on Defang before transitioning to ngrok for better accessibility. Cohere was used to build a context-based chatbot via VisionOS voice-activation.

Challenges we ran into

We ran into a few challenges understanding the differences between VisionOS view layouts, like with volumetric windows. We also had to learn how to use Reality Converter and 3D model design. Trying to make our 3D models interactive took a lot of time and teamwork to figure out, especially since the VisionOS is such a new platform! It also took a lot of practice to get used to the controls of the Apple Vision Pro…

Accomplishments that we're proud of

This was our first time developing on Vision Pro, and we successfully built the project using a lot of new tech! We integrated multiple purpose-built databases for file storage and metadata, implemented several AI/ML models, and managed to seamlessly bring everything together. Despite working with unfamiliar technologies, we created what we believe to be a truly game-changing application!

What we learned

We gained valuable experience in Swift and understanding of Vision Pro and the intricacies of AR/VR development, along with a ton of cool stuff about integrating different likes of technologies into a cohesive platform!

What's next for LockedIn (Vision Pro Application)

  • Upgraded 3D model generator with more customizability
  • Tracking user engagement, potentially adding some tools that test the user of page content that is read, for knowledge retention

Built With

Share this project:

Updates