TapToTalk

Needs highlighted
I need water ready to speak
the Call nurse card, raised at 8:27 AM PDT

Inspiration

For patients who are intubated in the ICU, something as simple as saying "I'm in pain" or "Call a nurse" can become extremely difficult. We wanted to explore how affordable hardware and AI could make basic communication more accessible.

What it does

TapToTalk is a prototype communication aid that lets people select predefined phrases using a single finger.

Patients navigate categories such as Needs, Pain/Comfort, People, and Yes/No with simple taps. Their selected phrase can then be spoken aloud, while nurse requests appear on a simulated caregiver dashboard.

How we built it

We combined a Raspberry Pi Pico and touch sensor with a Python bridge, WebSocket communication, and a browser interface.

MicroPython: Detects finger taps and holds.
Python and WebSockets: Connect hardware events to the patient interface.
HTML, CSS, and JavaScript: Provide large, accessible phrase buttons and a caregiver dashboard.
SQLite: Stores local request history.
Claude by Anthropic: Re-ranks predefined phrases based on permitted context.
Cartesia: Generates audio clips for spoken communication. The prototype supports local audio playback and is designed to fall back to keyboard controls when the sensor is unavailable.

Challenges we ran into

The biggest challenges were coordinating hardware and software, designing a reliable single-input navigation system, and keeping communication available when network services fail.

Accomplishments that we're proud of

We built a modular prototype combining accessible interface design, local audio playback, request logging, and AI-assisted phrase selection.

What we learned

We learned how to integrate microcontroller input with real-time web interfaces and how to design reliable fallback behavior for accessibility-focused applications.