FortWatch

Affordable watch to help deaf people sense important noises around them.

Comment

Inspiration

Nico's friend, Sofia, is deaf. She studies at Sierra College Nevada Campus, a small, very affordable community college in California. Talking about her disability, she said:

I feel such a lack of independence every time I want to go out in the city alone. I wish I had an Apple watch, but you know, it's out of budget. I wished I had a cheaper alternative specifically for sensing important noises.

When we came to Cal Hacks, we found an opportunity to help her and others in a similar situation.

What it does

It vibrates and shows the name of the sound when it detects relevant noises (such as doorbells, car horns, etc.). Besides, it shows the time, and syncs the data in real time with a dashboard to see some cool stats.

How we built it

  • A microphone on the wristband listens. Twice a second, a YAMNet-based model on the ESP32-S3 classifies the last second of audio, with no phone or internet needed.
  • It recognizes 10 alerts: alarm, car horn, glass breaking, shouting, baby crying, knock, doorbell, phone, dog barking and speech. Each threshold was tuned on the ESC-50 dataset.
  • The screen shows the alert full-screen with an icon and large text. The motor vibrates by urgency: three long pulses for safety, two short for attention, one short for low priority. A tap dismisses it.
  • Audio never leaves the wristband.
  • The case is 3D-printed with a snap-on back, and a power bank powers it.
  • Over Wi-Fi, each alert goes to a SpacetimeDB database hosted in Mana. A live dashboard there shows which sounds happened and when. The Mana agent wrote the code.
  • Claude Code co-authored more than 85% of the code, and also taught us about hardware in the process.

Challenges we ran into

  • Not all the components we needed were available to borrow at the venue, so we had to adapt our design and find creative alternatives.
  • Picking the right ML model
  • Finding an alternative when we did not have a suitable microphone module
  • Recovering from multiple 3D-printing failures

Accomplishments that we're proud of

We built a complete hardware prototype without having a dedicated hardware engineer on our team. Along the way, we overcame several challenges, listed above. Despite these obstacles, we successfully created a working end-to-end prototype that detects important sounds and sends visual and haptic alerts to the user.

What we learned

  • How to combine machine learning, Bluetooth communication, and embedded hardware into a complete end-to-end system.
  • Building hardware requires constant testing and iteration, as components do not always behave as expected.
  • How to break down unfamiliar problems, quickly develop the necessary skills, and work together as a team to achieve our goal.

What's next for FortWatch

  1. Make the design more compact, use a better material for the wristband, and use an on-device mic.
  2. Test it with Sofia and improve based on her feedback
  3. Make it as cheap as possible to produce. Based on our research with Claude Code, the components cost can get as cheap as $10.
  4. Raise money
  5. Produce it in mass
  6. Sell

Built With

Share this project:

Updates

Submission history