PulsePath

An assistive headwear piece that turns vision and sound into directional vibrations, helping people with vision or hearing loss find objects and turn toward certain sounds.

Comment

Inspiration

PulsePath started with people close to us.

Shravan’s friend Sam is deaf. When someone calls his name from across a room, he doesn’t hear it. People sometimes assume he’s ignoring them or being rude, when he simply has no way of knowing someone called him—or where they are.

For Max, the motivation is also personal: his grandmother has difficulty seeing. Her experience made us think about the everyday visual information we take for granted. A water bottle moved to another table, an unfamiliar room, or belongings mixed together on a crowded desk can turn a quick glance into a difficult search.

For blind and low-vision people, locating a specific object can involve searching by touch, remembering where it was last placed, or asking someone for help. We wanted to explore how a wearable could provide another option: ask for an object and feel which way to turn toward it.

The scale of vision loss makes this more than a problem close to home. The World Health Organization estimates that at least 2.2 billion people have a near or distance vision impairment. A study published in The Lancet Global Health estimated that 43.3 million people were blind in 2020, projecting that number to reach 61 million by 2050. These figures represent different levels of vision loss and different accessibility needs.

On the hearing side, WHO reports that 430 million people require rehabilitation for disabling hearing loss, with more than 700 million projected to need hearing rehabilitation by 2050.

What connected the experiences that inspired us was a missing sense of direction: where a voice came from, or where an everyday object was sitting.

That became our “aha” moment. Could the same simple cue—a vibration on the side you should turn toward—help with both?

Hearing and vision loss can also overlap. Conditions such as Usher syndrome affect both senses. This reinforced our interest in touch, while reminding us that different users need different controls and communication options.

Before building, we reached out to the Harvard Undergraduate Disability Justice Club. Their feedback highlighted challenges such as locating belongings in shared spaces and encouraged us to consider consistent cues, similar-looking objects, uncertainty, and user control.

Those ideas shaped PulsePath: a headband that turns direction into touch.

What it does

PulsePath supports two main uses: locating someone calling your name and finding an everyday object. Both communicate direction through vibrations.

Hearing assistance: locate someone calling your name

For deaf and hard-of-hearing users, PulsePath turns a spoken name into a tactile alert.

  • Recognize your name. Speech recognition identifies when someone calls the wearer’s name.
  • Locate the voice. Four microphones help estimate which direction the voice came from.
  • Feel where to turn. The band vibrates on that side. A voice from behind produces a vibration at the back of the head.
  • Recognize uncertainty. When microphone evidence is too weak to trust, the band gives a general attention pulse instead of guessing a direction.

Vision assistance: find an everyday object

For blind and low-vision users, PulsePath turns the camera’s view into directional guidance.

  • Ask for an object. Hold the Action button, say something like “find my water bottle,” and release it.
  • Confirm the target. The AI agent interprets the request and asks for confirmation before guidance begins.
  • Follow the vibrations. A forehead-mounted webcam looks for the target, and tactile cues guide the wearer toward it.
  • Use hot-and-cold feedback. Pulses change as the object becomes more centred and appears closer.
  • Know when guidance ends. A distinct “found” pattern signals completion. If the camera loses the target, the band gives a “lost” cue and pauses guidance.

Additional features: caregiver messaging and dashboard

  • Keep caregivers connected. Through the Photon API, PulsePath supports optional caregiver alerts over iMessage, with rate limits to prevent excessive notifications.
  • Ask for updates by text. A texting agent supports conversational requests such as “what did I miss?”
  • See the system working. A live dashboard displays microphone levels, camera output, and motor activity, with test controls and a STOP button.

What makes PulsePath different

  • Two tasks, one directional interface. Locating a voice and finding an object use the same basic instruction: feel the vibration and turn toward it. One wearable supports applications for deaf and hard-of-hearing users as well as blind and low-vision users.
  • Object guidance through touch. For blind and low-vision users, PulsePath translates the camera’s estimate of an object’s position into physical cues. The goal is to help answer “where is it?” without requiring the wearer to inspect a visual detection result.
  • Guidance without an audio stream. Directional feedback does not require hearing spoken instructions or reading a screen. For users who rely on environmental sounds, it avoids adding another audio stream to follow.
  • Affordable hardware. Our estimated prototype parts cost is approximately $40–$60, excluding the laptop used for processing. We built with an ESP32, inexpensive microphones, and vibration motors to keep affordability central to the design.
  • Support that includes caregivers. Accessibility also involves the people who help someone day to day. Optional iMessage alerts and conversational updates give caregivers another way to stay informed. PulsePath supports both the wearer’s independence and connection with their support network.
  • Four motors, eight directions. Front, right, back, and left each have a motor. Activating neighbouring motors together represents diagonals: front-right combines the front and right motors.
  • A focused interaction. The wearer requests an object and receives tactile guidance. The dashboard helps developers and judges inspect the system, but directional feedback comes through the band.
  • Feedback when it matters. Completion and lost-target cues are followed by quiet. We designed these transitions to avoid unnecessary continuous buzzing, and caregiver alerts are rate-limited to keep notifications manageable.
  • An honest “I’m not sure.” Uncertainty handling is part of the interface. The system distinguishes between having a useful direction and simply detecting that attention is needed.
  • Community-informed design. Feedback influenced concrete choices: consistent directions, distinct uncertainty cues, and target confirmation before object guidance begins.

How we built it

Our architecture follows three steps: sense, decide, and buzz.

Hardware and firmware

  • ESP32 controller: reads microphone levels and button events and controls the vibration motors.
  • Four MAX4466 microphones: positioned at the front, right, back, and left to compare sound levels.
  • Motor driver circuits: each vibration motor has its own transistor channel, with a flyback diode and capacitor.
  • Separate motor power: two AA batteries supply the motors, with a shared ground to the ESP32.
  • Camera and Action button: a forehead-mounted USB webcam provides video, while the button enables hold-to-talk requests.
  • USB serial connection: the ESP32 streams measurements to the laptop, which sends back pulse commands. Pulses are time-limited, and the firmware stops the motors on a STOP command or connection loss.

Speech and sound direction

  • We used faster-whisper to recognize the wearer’s name and transcribe object requests.
  • The laptop buffers microphone measurements and aligns the ESP32’s clock with speech timestamps, allowing us to examine the moment the name was spoken.
  • We correct for differences in microphone sensitivity, compare front against back and right against left, and estimate a direction.
  • Noise filtering and confidence checks determine whether to produce a directional cue or a general attention pulse.
  • A sound-alike check helps reject similar names that should not trigger an alert.

Vision and AI

  • Ultralytics YOLO detects common objects, with YOLO-World / YOLOE supporting open-vocabulary detection.
  • A tracker smooths object position and bounding-box size, keeping small frame-to-frame changes from producing distracting vibration changes.
  • A LangChain Deep Agent powered by Anthropic’s Claude interprets requests and coordinates the search, with a camera “scout” and a “memory-keeper” for previously seen objects.
  • The agent pauses for target confirmation before guidance begins.
  • The immediate tracking and haptic loop stays in plain Python, so each directional update does not have to wait for a language model.

Caregiver messaging

  • We integrated the Photon API to support optional caregiver alerts through iMessage.
  • A texting agent supports conversational updates, including requests such as “what did I miss?”
  • Rate limits help prevent excessive notifications and keep updates manageable.

Team workflow

  • We divided work into software integration, firmware, vision and speech, and the hardware protocol.
  • We coordinated through Band, with AI coding agents supporting different lanes.
  • A shared protocol defined pin assignments, serial messages, and cue behaviour.
  • Simulated hardware and recorded microphone inputs let software development continue while the physical band was being assembled.

Challenges we ran into

  • Microphones that disagreed. The microphones had different sensitivities, so the same sound produced different readings. We calibrated them individually. We also caught left and right microphone wiring that was reversed relative to the documented pin map, causing mirrored direction estimates.
  • Motors interfering with microphones. Our feedback hardware introduced noise into our sensing hardware. We measured during quiet gaps and skipped microphone windows while motors were buzzing.
  • Two clocks describing one moment. Speech timestamps came from the laptop, while microphone measurements came from the ESP32. We had to align them so a recognized name was paired with the correct sound measurements.
  • Unstable object guidance. Detection boxes shifted between frames even when an object was still. Tracking and smoothing helped prevent cues from rapidly switching directions.
  • Shared hardware resources. Speech, vision, the agent, and the dashboard needed to work together without competing for the same devices. Leaving the Arduino Serial Monitor open could prevent the application from accessing the serial port.
  • Library compatibility. GPU-backend crashes on macOS led us to run vision on the CPU for the demo. We also pinned an audio dependency to keep speech recognition working reliably.
  • Moving from simulation to a wearable. Software that behaved correctly with recorded data still needed real-world calibration for microphone placement, motor orientation, and pulse timing.

Accomplishments that we're proud of

  • A working physical prototype. Microphones, camera input, speech recognition, object detection, and vibration motors communicate through one integrated system.
  • Two core features on one band. Name-direction alerts and object-finding guidance share a consistent tactile interface.
  • A connection beyond the wearable. Photon-powered caregiver messaging extends the project to the people supporting the wearer.
  • Promising early direction results. The system identified 15 of 16 directions correctly across two worn-band rounds using a recorded speech clip.
  • Improved object tracking. Object-side accuracy increased from 87.3% to 93.3% across four recorded evaluation clips after tracker tuning.
  • More stable feedback. Smoothing reduced rapid motor-direction changes in our steady-object evaluation.
  • Automated verification. The vision branch’s test suite grew to more than 300 passing tests.
  • Feedback that changed the implementation. Community input shaped uncertainty handling, consistency, and user confirmation.

These results come from limited prototype evaluations. Live speech in noisy rooms and testing with intended users remain important next steps.

What we learned

  • Start with one complete working path. Getting one input to produce one useful vibration taught us more than polishing disconnected components. We improved calibration and timing after the basic interaction worked.
  • Agree on interfaces early. A shared hardware protocol helped teammates work independently and prevented hours of integration confusion.
  • Simulate before you solder. Recorded inputs and a simulated band kept software development moving while the hardware was still taking shape.
  • Hardware can fail quietly. Swapped pins, wiring issues, and mismatched microphone gains often produced plausible but incorrect answers. Testing one component at a time was essential.
  • Responsiveness needs its own architecture. AI can coordinate a search, but immediate directional feedback needs a fast, predictable loop.
  • Uncertainty is part of usability. An incorrect direction can be more confusing than a clear indication that the system is unsure.
  • Independence and connection can work together. Direct guidance supports the wearer, while optional messaging helps keep caregivers informed. Both deserve thoughtful controls and manageable feedback.
  • Accessibility requires continued collaboration. Community feedback improved our design, but it does not replace hands-on testing with intended users. Comfort, communication preferences, and useful cue patterns need to be evaluated with the people who would wear the band.

What's next for PulsePath

  • Test with intended users. Evaluate comfort, cue clarity, and usefulness with blind, low-vision, deaf, and hard-of-hearing participants.
  • Refine caregiver updates. Work with wearers and caregivers to understand which updates are useful, when they should be sent, and how users want to control sharing.
  • Offer more ways to make requests. Add alternatives to spoken commands and explore controls with deafblind users.
  • Move beyond the laptop. Shift processing to a phone and connect the band over Bluetooth.
  • Handle similar-looking objects. Ask for clarification when multiple objects match a request.
  • Track head movement. Add an IMU to improve how guidance responds as the wearer turns.
  • Refine the wearable. Build a softer, rechargeable band and publish a reproducible parts list.

The moments that inspired PulsePath are small but meaningful: knowing someone is calling you, locating something nearby, and staying connected to someone who cares. We want to make those moments easier.

Built With

Share this project:

Updates

Submission history