SENSEA

• 

  SENSEA Logo

• 

• 

• 

• 

• 

Inspiration

Every student should be able to explore their campus with greater independence. For students with disabilities, especially blind and low-vision students, unfamiliar paths, confusing building names, and inaccessible navigation interfaces can make everyday trips more difficult. We created SENSEA to make campus walking more convenient through voice-first navigation, personalized preferences, and on-device visual assistance. Our goal is to help students access the places and experiences that make campus life meaningful.

What it does

SENSEA helps students search for campus destinations, compare walking routes, and receive spoken GPS guidance. Users can search by building name or by departments, libraries, and dining locations inside a building. The app supports voice commands, recent destinations, saved places, and personalized settings. Its camera processes images on-device to identify objects and provide experimental attention cues. On Android, expanded analysis also explores pedestrian-signal observations. SENSEA also supports optional environmental sound measurements, laying the foundation for a campus noise map. Camera observations and noise readings provide additional information; they do not establish that a path or crossing is safe.

How we built it

We built the mobile app with React Native, Expo, and TypeScript, with a Python FastAPI backend. Google Maps displays the campus map, while the Google Routes API provides walking-route alternatives. Supabase handles authentication, user preferences, saved destinations, route history, and consent-based noise contributions. We collected building and facility information from UW–Madison’s official campus map to support more useful destination searches. For visual assistance, we use VisionCamera and TensorFlow Lite for local object detection. Android adds native Kotlin components, an ONNX model, and ML Kit OCR for expanded street-object and signal analysis. We combined these components with speech recognition, spoken feedback, GPS filtering, and vibration cues.

Challenges we ran into

One major challenge was making camera analysis work consistently across iOS and Android. Camera lifecycle management, native dependencies, and resource cleanup required careful debugging, particularly when stopping and restarting analysis. We also had to coordinate speech recognition, spoken navigation, and microphone measurements without letting them interfere with one another. GPS inaccuracies and network failures introduced additional challenges for reliable guidance. Another challenge was distinguishing available information from verified information. Building coordinates do not identify accessible entrances, noise does not reliably measure crowd size, and object detection does not establish distance or crossing safety. We designed the app to communicate these limitations instead of presenting uncertain observations as facts.

Accomplishments that we're proud of

We brought destination search, user accounts, route comparison, GPS guidance, and camera observations together in one working mobile app. We integrated 219 UW–Madison building map entries and 589 facility entries, allowing students to search for places beyond a building’s official name. We also simplified destination selection, added personalized recent places, and implemented recovery from common camera and network failures. We are proud of building a foundation that combines useful campus information with on-device processing and optional, consent-based data contributions.

What we learned

We learned that accessibility involves much more than adding voice output. Clear interaction flows, fewer confirmation steps, understandable error messages, and alternatives to voice input all matter. We also learned that navigation depends on the quality and freshness of its underlying data. A route-ranking algorithm is only useful when it can distinguish verified conditions from missing information. Finally, we learned to separate successful software tests from real-world performance. Camera recognition, microphone behavior, and accessible navigation still require testing on physical devices and feedback from the students who would use them.

What's next for SENSEA

Our next goal is to connect university building APIs and available indoor maps so SENSEA can help students navigate inside buildings as well as between them. We also want to make SENSEA useful to the wider campus community. Through optional contributions, students could help build a time-based sound map showing where campus is quieter or more active. This could help students find study spaces, explore lively areas, or discover possible campus activity. Confirming specific events would require additional event information. As this dataset grows, we aim to improve personalized walking recommendations. During the day, we want to prioritize comfortable routes using verified path conditions and sound preferences. At night, we want to explore preferences for well-lit routes with evidence of nearby activity, with the aim of reducing isolation. Noise alone cannot establish that people are present or that an area is safe. We plan to combine it with verified lighting and pedestrian information, validate recommendations with users, and communicate uncertainty clearly.

Built With

• expo.io
• fastapi
• google-ml-kit
• google-routes-api
• kotlin
• onnx-runtime
• postgresql
• python
• react
• supabase
• tensorflow
• typescript
• visioncamera