Doorway

Inspiration

As a freshman at UCF we all had difficulties navigating through campus and finding the rooms for classes, clubs, and meetings. That's why we built Doorway to solve our problems with campus navigation.

What it does

Doorway is an AR app that gives turn-by-turn directions inside campus buildings, where GPS stops working. It works the way Google Maps does outdoors.

Open the app, pick where you're going, and Doorway draws a glowing route on the floor through your phone's camera, all the way to the room. An outdoor campus map connects them: tap the other building to load its floor.

Finds where you are indoors, without GPS. It combines AR tracking, step counting and the building's walls to work out your position. It also reads room-number signs ("121", "107A") through the camera to pin down exactly where you are.
Guides you in AR. A route on the floor and a turn banner tell you where to go next. It recalculates if you go a different way.
Talks with you. Say "take me to room 107" or "where's the bathroom?" to set a route. Doorway reads each turn aloud, so you can keep your eyes up instead of on your phone.
Lets you browse the building. Search or scroll a directory of rooms, bathrooms, elevators, stairs and entrances. A floor map you can zoom and a minimap show where you are and where you're headed.
Works without a connection. Floor plans, routing and sign reading all run on the phone.

How we built it

The app. Doorway is built in Unity 6 with C#, using AR Foundation and ARCore on Android. ARCore tracks the phone's movement and senses depth, so the app can tell walls from floor. A Java plug-in brings in Google's ML Kit text recognition, which reads room-number signs in the camera feed, fully on the phone. Voice commands and spoken directions use Android's built-in speech recognizer and text-to-speech. We call both directly from C#, with no extra plug-in.

Finding you indoors. With no GPS inside, our localizer combines several weaker signals into one position. It uses AR tracking, steps from the phone's step sensor, a match between the walls ARCore sees and the walls on the floor plan, and every room sign the camera reads. A router finds the shortest path through a walkable grid of the floor. A navigator turns that path into turns, then reroutes or re-aligns when you drift.

Turning floor plans into maps. We started from floor plans traced by Knight Hacks and archived UCF floor plans. Python pipelines with OpenCV, NumPy, SciPy, Shapely and Pillow pull walls, rooms, doors and walkable space out of the images. They then produce the floor data and themed map images the app loads. The outdoor campus map comes from OpenStreetMap, downloaded through the Overpass API and lined up with the floor plans.

Training a voice model. We generated labelled voice commands from the building's own rooms. We added more natural wordings using Hugging Face Inference Providers, then fine-tuned Google's FunctionGemma 270M on Hugging Face Jobs to turn speech into app commands. A rule-based parser in the app handles commands instantly and offline.

Testing with real walks. We walked the buildings with the app, which logs every position it estimates. We then replayed those walks and generated synthetic ones in Unity to measure positioning error and tune the localizer. The logs go into Databricks, where tables and maps show how far off the app was and where on each floor it struggles.

Keeping it working. GitHub Actions runs our pytest suite on the data pipelines. Through GameCI, it also runs Unity's tests and builds a test APK.

Challenges we ran into

We faced lots of difficulties figuring out how to make our AR tracking more accurate. One of the biggest causes to this problem was the outdated floor plans that UCF made publicly available. Due to this constraint, our AR tracking would have difficulties navigating through furniture, new pathways, and people.

Accomplishments that we're proud of

We're happy to have learned AR tracking for the first time and implimented it into an application that serves a very useful purpose to students, faculty and others on campus. not just for UCF but for other institutions in the future

What we learned

We learned to perform more efficiently in a group of 3 throughout the course of 36 hours to completely start from zero and end up with a working proof of concept. We also combined a lot of technologies such as Unity, AR Foundation, ARCore, Google's ML Kit, Android's speech recognition and text-to-speech, Python with OpenCV and NumPy, OpenStreetMap, Hugging Face with Gemma, and Databricks into one app, and learned how to make them all work together.