FirstHour

Inspiration

Many falls at home happen in the bathroom. People rarely carry a phone in there, and nobody wants a camera there. A fall pendant only helps if the person is wearing it and can press it. We wanted a device that notices the fall, asks the person how they are, tells someone who can help, and lets that person in.

What it does

FirstHour is a camera-free fall-response node for the bathroom. An ESP32-S3-BOX-3 senses, a Raspberry Pi 5 running QNX decides, and a laptop server handles the AI and messaging.

The thud. The BOX-3 streams its microphone and accelerometer 20 times a second. The Pi opens an incident only when a sound spike and an impact spike happen within 400 ms of each other. Before any network call, the Pi sounds the alarm and the screen shows TIME DOWN 00:01.
The voice. A calm voice asks, "Are you okay? Say 'I'm fine' or 'I need help.'" A short fixed set of triage questions follows. The Pi transcribes her spoken answers with whisper.cpp running on QNX. Only a clear "I'm fine" cancels the incident. Silence escalates it.
The text. The caregiver gets an iMessage triage card: time down, what the person falling said, and whether they're on a blood thinner (or any other disease based on the profile setup, we have setup blood thinner for demo purposes). The first alert asks "How soon can you be here?". When the caregiver replies "on my way," the reply is read aloud to the person, and the one-time door code is texted to that caregiver only. If nobody replies within 20 seconds, the system places a voice call to the assigned care giver.
The door. UNLOCK <code> opens the door only if every rule passes: an open incident, an enrolled number, the correct one-time code, fewer than 3 wrong tries, and a signed, fresh, single-use command. The Pi then checks on its own that the command names its own open incident. A stranger who has the right code is refused. The latch is simulated (we had no servo available to us to actually show a door unlocking), and it relocks itself after 30 seconds.

After the triage questions, an ElevenLabs companion agent keeps her company and decides what is worth texting the caregiver. A live dashboard shows the clock, the triage record next to her exact words, the conversation, and every unlock attempt with its reason.

How we built it

Sensor hub: Arduino-ESP32 firmware on the ESP32-S3-BOX-3. It uses the board's built-in IMU, dual microphones, speaker and LCD, so there's no wiring. The microphone and the alarm each run as their own FreeRTOS task.
Edge: Python on QNX 8 on a Raspberry Pi 5. A supervisor runs three processes at different QNX priorities: the safety process (21), the network uplink (10), and the whisper.cpp server (8). It restarts any process that dies. The uplink signs nothing, so every signature check happens in the safety process.
Backend: FastAPI. It runs the incident state machine, rule-based triage refined by StepFun step-5-preview, caregiver coordination, the unlock rules and HMAC signing, cached ElevenLabs speech, and Sentry tracing.
Messaging: a Node and TypeScript gateway on the Photon 2.0 beta SDK, for iMessage and SMS out and replies in. Voice calls go through an ElevenLabs agent on a Photon number.
Dashboard: static HTML and JavaScript, plus a caregiver-phone simulator that posts to the same endpoint Photon uses.

Challenges we ran into

No USB-serial driver on QNX. The Pi couldn't see the BOX-3 as a serial port, so we read it over raw USB with pyusb and libusb.
A board reset crashed the reader. Pressing reset on the BOX-3 aborted the native USB code before Python could catch anything. We moved USB into a child process that restarts on its own. The incident, clock and alarm now survive a reset.
No microphone on the Pi. The webcam's microphone failed under QNX audio. The laptop now records each answer, and the Pi fetches the clip with a signed request and transcribes it locally.
whisper.cpp stalled on 4 threads. On the 4-core Pi it took 272 ms per decoded token. On 3 threads it took 9.5 ms.
The model was slow and its speed swung. StepFun triage took 5 to 38 seconds. We moved it off the critical path. An instant rule-based record drives the alarm and the alert, and the model refines it in the background.
Replies were being dropped. Photon gives each caregiver a different shared line, and our gateway only listened on one of them. Sentry tracing helped us find it.

Accomplishments that we're proud of

The Pi decides. Detection, the alarm, the clock, signature checks and the final unlock decision all run on the Pi. The cloud can ask, but it can't open the door.
It recovers from crashes on real hardware. When we killed the uplink in the middle of an incident, it reconnected in under 2 seconds. When we killed the safety process, it resumed the same incident in under 4 seconds.
The unlock chain ran end to end on hardware. A stranger with the right code was refused, the caregiver was accepted, the Pi verified the signed command, and the BOX-3 opened and relocked.
Live iMessage worked. A real reply from a phone reached the backend in 3.5 seconds.
The AI never makes a safety decision. The questions are fixed, medications come from the profile, unknown answers stay unknown, and the model can only add worrying findings, never reassurance.

What we learned

Keep the model next to the critical path, not in it. Run the rules first, let the model refine in the background, and let the rules win.
A real-time OS proves its worth when you can kill the network process during an incident and the alarm doesn't skip a beat.

What's next for FirstHour

Full calibration with actual human falls. For the hackathon purposes, we used knock on a table to simulate a human thud but the actual real world use would require us calibrating the sensors to only calibrate and detect actual human fall thuds.
A real latch, driven by a QNX resource manager that only the safety process can write to.
A paramedic handoff sheet.
Testing with clinicians and older adults. Until then, this is a prototype, not a medical device.