-
-
App main page
-
Locked passive hearing notification - siren
Inspiration
Our primary early warning senses come from sound. Hearing glass breaking lets us know theres a break in. Hearing an alarm lets us know theres a fire. Millions of deaf and hard-of-hearing people are susceptible to risks, simply due to the fact that they cannot detect the warning.
What it does
SoundSight turns an iPhone into a 360° sound radar It listens to the room, recognizes important sounds (smoke alarm, broken windows, dogs barks, etc.), shows where each one is coming from, and tells you how urgent it is.
How we built it
Spatial audio was implemented by leveraging Apple's spatial microphone recording. The Edge AI classification works in two parts. The first part is using Apple's built-in sound recognition. Google's YAMNet operates on top of that for background alerts. Stack: Swift and SwiftUI, AVFoundation, Core ML, SoundAnalysis, Core Motion, Core Haptics, ActivityKit, UserNotifications and XcodeGen.
Challenges we ran into
Apple's strict constraints on ios development were pretty rough to deal with. The original idea for spatial audio had to be scrapped due to being unable to access individual raw microphone readings. There were also a lot of false alarms, with simple tapping of the phone setting off the radar.
Accomplishments that we're proud of
- 360° direction from a single phone with no extra hardware, ±10° in prod.
- Edge AI that keeps working even when phone isn't on, allowing 24/7 background alerts.
- Fully private: audio never leaves the device. ## What we learned How spatial audio works and a lot more about IOS development. ## What's next for SoundSight Apple watch alerts, real testing with deaf & hard-of-hearing individuals, and custom model.
Log in or sign up for Devpost to join the conversation.