Inspiration

Physical buttons are disappearing. Microwaves, thermostats, self-checkouts and airport kiosks are now flat glass, and for someone who can't see, flat glass has nothing to feel. Over 7 million Americans live with vision loss that glasses can't fix. Navigation tools can get a blind person to the right room, but nothing gets their hand onto the right control. We call that gap the last arm's length.

What it does

Nudge is an iPhone and Apple Watch app that guides a blind or low-vision person's finger to the right button on a touchscreen.

  1. Point the phone at the screen and say what you want, for example "cook for 90 seconds".
  2. Nudge reads every button on the screen and plans the steps: 9, 0, Start.
  3. It tracks your fingertip and guides it with stereo tones, voice and taps on your wrist.
  4. After each press it re-reads the screen to check the press worked, then moves to the next step.

There is no setup per device and no hardware to install on the appliance.

How we built it

  • Native Swift app for iPhone, with a watchOS companion app for wrist haptics.
  • ARKit and LiDAR lock the screen's position in 3D so button locations stay put as the phone moves.
  • Apple Vision reads the button labels and tracks the hand, all on the phone.
  • An AI planner turns the spoken goal and the list of buttons into an ordered set of presses.
  • Stereo audio, speech and haptics tell the user which way to move and when to tap.
  • Velocity AI (Growth Signals) for market research and concept assessment. We ran its "Assess a market opportunity" guide and a research chat on Nudge.

Camera frames never leave the phone. Only the button labels are sent to the planner, as text.

Challenges we ran into

  • Finger tracking is the hardest part. Tracking drops when the palm leaves the camera's view. We built a record-and-replay tool so we could reproduce the phone's tracking on a Mac and measure every fix before shipping it.
  • Light and framing. Hand tracking failed often in a dim room, so the app now coaches the user on where to aim the phone.
  • Keeping buttons in place. Early versions lost the button positions whenever the phone moved, which is why we moved to ARKit and LiDAR.

What we learned

  • Reading a screen is the easy part. Guiding a hand to a point in space, and confirming the press, is the real problem.
  • Velocity AI changed our go-to-market. It showed that home appliances have the weakest accessibility mandate, while airports and airlines have the clearest requirement to buy, so that is where we would sell first.
  • Measuring beats guessing. The replay tool saved us from fixes that only looked like they worked.

What's next for Nudge

  • Test with blind and low-vision users. So far it has only been tested by the two of us, on a microwave panel shown on a screen.
  • Add a LiDAR fallback for finger tracking and try real appliances and kiosks.
  • Go beyond touchscreens to anything you touch: lift buttons, door handles, light switches, card readers.
  • Move the camera to a wearable, with guidance on the wrist, so it works hands-free.

Nudge is a hackathon prototype. It is not clinically validated or certified.

Built With

Share this project:

Updates

Submission history