Inspiration

After surgery, or with a stroke, ALS, or a breathing tube, speaking can be exhausting or impossible. Reaching a call button can be just as hard. Many of these patients can still make a small, deliberate hand shape. We wanted a call system that works from that, and that keeps video of a patient in a hospital bed off the cloud.

What it does

CareGesture is a bedside communication prototype that turns simple hand gestures into caregiver requests.

  • ✋ Open palm → assistance
  • ✌️ Two fingers → water
  • ✊ Closed fist → room comfort

A camera watches a defined gesture area. A hand-landmark model and a classifier recognize the gesture, and a confirmation step makes sure it was deliberate. The request then appears on a nurse dashboard, where it can be acknowledged and resolved. The patient's screen shows the status, so they know someone saw their request. A room-comfort request also includes a snapshot of the room's temperature and humidity from a Modulino Thermo sensor, so the nurse knows what the room is like before walking in.

How we built it

  • Edge AI: We used an Edge Impulse model to collect and classify hand gesture data, and then trained on our target hardware, the Qualcomm-powered Arduino UNO Q.
  • Backend: FastAPI with a request store, live updates over a streaming endpoint, and a documented API. It has a mock mode for development without hardware.
  • False-alert protection: a deterministic gesture state machine. A gesture must hit a minimum confidence and be held steadily for 1.5 seconds. Each hold fires exactly one request, and the hand must return to neutral before another can fire. A camera stall or clock jump discards a hold in progress. The state machine uses only prediction timestamps, never the wall clock, so it is fully unit-tested.
  • Frontend: a launcher, a patient guide, and a nurse view with status cards. It has configurable sound alerts (patterns, repeats, mute, volume, speech).
  • Sensor: a Modulino Thermo reading, attached to comfort requests as an environment snapshot.

Challenges we ran into

  • Making recognition reliable: A call system that fires on a stray movement is worse than none, so we built the hold-and-release state machine instead of triggering on a single frame.
  • Model reliability: We tried a few different model architectures and methods before deciding on the best one. Starting off, we used the MediaPipe hand landmarker model. Then, we trained a small TensorFlow classifier on our own collected landmark data, and exported it to TFLite and Keras for on-device inference. Eventually, we decided to use an Edge Impulse model due to its higher reliability and ease of data collection and inference.
  • Collecting enough varied gesture data in a hackathon, and keeping the model accurate across different hands and lighting.
  • Latency on Windows: localhost resolved to IPv6 first and added about 2 seconds per request, which broke our hold timer until we switched to 127.0.0.1.
  • Coordinating four people's parts (camera, backend, frontend, sensor) around one API contract.

Accomplishments that we're proud of

  • An end-to-end loop: gesture → confirmed request → nurse action → status back to the patient.
  • Inference runs locally, so no patient video leaves the room.
  • A tested false-trigger protection layer.

What we learned

Accessibility tools have to prioritize reliability over cleverness. Confirmation design matters as much as model accuracy. We also learned a lot about shipping a small model to edge hardware.

What's next

  • Customizable gestures per patient, since not every patient can make the same hand shapes
  • More request types and priority levels
  • Integration with real nurse-call systems
  • Clinical validation with patients and nurses

Note: CareGesture is an accessibility demo. It is not a medical or nurse-call device. The Thermo reading is room air only, never body temperature.

Built With

Share this project:

Updates

Submission history