Inspiration
We all know how difficult to find things we've misplaced in a room, even when they're hiding in plain sight. That's with good vision. Imagine then, what it's like for someone who can't see in the first place. Misplacing an item could mean ages of feeling around the room, a frustrating and time-consuming endeavour.
What it does
InSight allows people with low- or no vision to locate objects in a room using entirely on-device AI. Users start the scanner, put in a natural-language request (through typing or voice), and show their phone the room. InSight uses on-device Foundation Models to process what exactly they want to find, and an on-device model running in Core ML to pick the object out from a RealityKit camera feed.
The app provides haptic and voice guidance to the item, which intelligently guides the user in regards to orientation and distance (determined via LiDAR) to the object.
How I built it
I built this app in Swift, leveraging SwiftUI, RealityKit, Foundation Models, Core ML, and more. Achieving accessibility and offline-only functionality were my main priorities. There was a healthy mix between manual coding and AI assistance. I'm used to programming entirely by hand, so used AI for ideation on implementation, learning, and simple/tedious work here.
Challenges I ran into
Given this was my first time integrating AI models like this, or using AR as a whole, I faced challenges learning what I needed to to get started. I needed to figure out how to integrate these models, what the relevant app architecture should look like, and how to mess with a 3D world as a whole. It was deeply intimidating at first but I'm proud of what I've been able to learn and achieve.
Accomplishments that I'm proud of
This was my first app that made use of AI models to this extent. My past experience had only been having Foundation Models summarise some text here and there. I'm incredibly proud of how I was able to integrate a third-party model onto an iPhone app and actually have it run at high performance locally. I'm very happy with how I was able to combine multiple AI models into one product - like the YOLO26 model for object detection, but also Foundation Models to process user requests.
This was also my first AR app. I'm proud of how I was able to control the 3D space, and use the iPhone's special features like the LiDAR sensor for depth accuracy.
What I learned
I learned just how powerful iOS devices are for local, on-device AI. I'm genuinely impressed my phone was able to run YOLO26 alongside AR, Foundation Models, and more at a steady 60 fps. I also learned a bit more about AR itself, and overall what it's like building with accessibility as the top priority.
What's next for InSight
I'd like to polish and submit InSight to the Swift Student Challenge in the future. By then, it will include Create ML functionality that allows users to scan in their own custom items and have the app be able to pick them out of a scene too - helping differentiate things like "my water bottle vs everyone else's".
Log in or sign up for Devpost to join the conversation.