Inspiration
After surgery, or with a stroke, ALS, or a breathing tube, speaking can be exhausting or impossible. Reaching a call button can be just as hard. Many of these patients can still make a small, deliberate hand shape. We wanted a call system that works from that, and that keeps video of a patient in a hospital bed off the cloud.
What it does
CareGesture is a bedside communication prototype that turns simple hand gestures into caregiver requests.
- ✋ Open palm → assistance
- ✌️ Two fingers → water
- ✊ Closed fist → room comfort
A camera watches a defined gesture area. A hand-landmark model and a classifier recognize the gesture, and a confirmation step makes sure it was deliberate. The request then appears on a nurse dashboard, where it can be acknowledged and resolved. The patient's screen shows the status, so they know someone saw their request. A room-comfort request also includes a snapshot of the room's temperature and humidity from a Modulino Thermo sensor, so the nurse knows what the room is like before walking in.
How we built it
- Edge AI: We used an Edge Impulse model to collect and classify hand gesture data, and then trained on our target hardware, the Qualcomm-powered Arduino UNO Q.
- Backend: FastAPI with a request store, live updates over a streaming endpoint, and a documented API. It has a mock mode for development without hardware.
- False-alert protection: a deterministic gesture state machine. A gesture must hit a minimum confidence and be held steadily for 1.5 seconds. Each hold fires exactly one request, and the hand must return to neutral before another can fire. A camera stall or clock jump discards a hold in progress. The state machine uses only prediction timestamps, never the wall clock, so it is fully unit-tested.
- Frontend: a launcher, a patient guide, and a nurse view with status cards. It has configurable sound alerts (patterns, repeats, mute, volume, speech).
- Sensor: a Modulino Thermo reading, attached to comfort requests as an environment snapshot.
Challenges we ran into
- Making recognition reliable: A call system that fires on a stray movement is worse than none, so we built the hold-and-release state machine instead of triggering on a single frame.
- Model reliability: We tried a few different model architectures and methods before deciding on the best one. Starting off, we used the MediaPipe hand landmarker model. Then, we trained a small TensorFlow classifier on our own collected landmark data, and exported it to TFLite and Keras for on-device inference. Eventually, we decided to use an Edge Impulse model due to its higher reliability and ease of data collection and inference.
- Collecting enough varied gesture data in a hackathon, and keeping the model accurate across different hands and lighting.
- Latency on Windows:
localhostresolved to IPv6 first and added about 2 seconds per request, which broke our hold timer until we switched to127.0.0.1. - Coordinating four people's parts (camera, backend, frontend, sensor) around one API contract.
Accomplishments that we're proud of
- An end-to-end loop: gesture → confirmed request → nurse action → status back to the patient.
- Inference runs locally, so no patient video leaves the room.
- A tested false-trigger protection layer.
What we learned
Accessibility tools have to prioritize reliability over cleverness. Confirmation design matters as much as model accuracy. We also learned a lot about shipping a small model to edge hardware.
What's next
- Customizable gestures per patient, since not every patient can make the same hand shapes
- More request types and priority levels
- Integration with real nurse-call systems
- Clinical validation with patients and nurses
Note: CareGesture is an accessibility demo. It is not a medical or nurse-call device. The Thermo reading is room air only, never body temperature.
