SpatialVision

AR

Inspiration

Have you ever looked at a 2D photo on your phone and wished you could bring the subjects out of the screen and into your physical environment? We live in a spatial computing era, yet the billions of photos in our camera rolls are trapped behind flat glass. We wanted to bridge the gap between our digital memories and our physical spaces. Our inspiration for SpatialVision was to create an instantaneous, magical experience where anyone could turn their ordinary 2D photos into interactive, 3D spatial cutouts that live alongside them in the real world.

What it does

SpatialVision is an Augmented Reality application that transforms any flat photo into a mixed-reality experience.

Select: You choose any photo from your camera roll. Extract: The app instantly analyzes the image, identifies the main subject (a person, a pet, an object), and magically strips away the background. Project: By scanning your physical room with your camera, the app detects physical surfaces (floors, tables, walls) and projects the isolated subject as a 3D billboard into your environment. You can pinch to scale the cutout, drag it around your room, and capture mixed-reality screenshots to share with friends!

How we built it

We built SpatialVision entirely as a native iOS application using Apple's cutting-edge frameworks to ensure maximum performance and seamless integration:

Swift & SwiftUI: We used SwiftUI to build a beautiful, modern, glassmorphism-style overlay interface that feels native to the ecosystem. Apple Vision Framework: To achieve the instant background removal without relying on slow or expensive third-party cloud APIs, we utilized the new iOS 17 Vision Framework (VNGenerateForegroundInstanceMaskRequest). This processes the image entirely on-device, ensuring privacy and blazing-fast speeds. ARKit & RealityKit: We used ARKit for robust environmental understanding (horizontal/vertical plane detection and depth occlusion) and RealityKit to dynamically generate 3D meshes, apply our extracted 2D image as a TextureResource, and handle spatial gesture interactions.

Challenges we ran into

The biggest technical hurdle was bridging the gap between 2D image processing and 3D spatial rendering.

Dynamic Material Generation: Converting a standard UIImage into a transparent RealityKit SimpleMaterial required careful manipulation of CoreGraphics (CGImage) and aspect ratio calculations to ensure the 3D plane didn't stretch or warp the extracted photo. On-Device Machine Learning: Processing high-resolution images through the Vision framework required us to optimize memory usage and handle the extraction asynchronously using Swift Concurrency (async/await) so the UI wouldn't freeze while the neural engine stripped the background.

Accomplishments that we're proud of

We are incredibly proud that we achieved 100% on-device processing. By leveraging the native iOS 17 Vision framework, we didn't have to rely on external APIs like Remove.bg. This means our app works instantly, requires no internet connection for the heavy lifting, and respects user privacy. We're also proud of the intuitive RealityKit gesture integration—being able to organically grab, rotate, and scale a 2D memory in 3D space feels genuinely futuristic.

What we learned

We gained a massive appreciation for spatial computing design. We learned how to seamlessly pass data between SwiftUI (declarative UI), Vision (machine learning), and RealityKit (3D rendering). We also learned the intricacies of ARKit's raycasting and anchor system, discovering exactly how to tether digital coordinates to physical planes accurately.

What's next for SpatialVision

This hackathon was just the beginning! In the future, we plan to implement:

Multi-Subject Extraction: Allowing users to pick out multiple different subjects from a single photo to create a 3D diorama. Spatial Audio: Attaching ambient sounds to the 3D cutouts based on the photo's metadata. Persistence: Saving the physical coordinates of where you left a 3D cutout in your room, so when you open the app the next day, the memory is still sitting right where you left it!