ApexNurse

Individual Clinic Report Page
All Clinic Reports Page

Our Inspiration

ApexNurse all started with Justin's Aunt Sue. Sue is an ICU nurse, and works hard making sure that the public is taken care of. She often works super long and intense shifts, which leave her really tired when she's off of the job. She's not alone either. While doing research, we found that most nurses work similar agonizing shifts, with most hospitals being understaffed, leaving patients waiting longer, and more likely to be misdiagnosed. 200000 nurses switched fields in 2023 and there is a nursing pandemic in the country right now. All of this led us to wanting to make a solution to this problem.

What it does

The multimodal AI monitoring system, integrated into a nurse's face mask, continuously captures video and audio during patient interactions to detect medical emergencies, security threats, and anomalies (e.g., fainting, injuries, distress calls). Alerts are intelligently routed to appropriate specialists such as security with arrival time estimates. It includes the Jeff AI voice agent, which uses recursive RAG with hospital-specific context to assist nurses with queries and tasks. It also features automatic translation capabilities. Post-session, detailed clinical reports are generated from the recordings and uploaded to a PostgreSQL database, accessible via a mobile app for comprehensive documentation and review.

How we built it

We built a Raspberry Pi–powered smart face shield with a mic, rpi-camera, and transparent display mounted on 3D-printed hardware. A Python backend uses OpenCV and PyAudio for real-time capture, sending multimodal data to GPT-4o-mini Vision and Whisper to detect medical emergencies or aggression and trigger immediate staff alerts. Saying the wake word “Jeff” activates a Siri-like assistant that supports Q&A, two-way translation, and guidance through a recursive RAG system with an expanded clinical context window. After each session, GPT-4o auto-generates a clinical report from the event log and transcript and uploads it to Supabase, where an Expo mobile app displays vitals, notes, and audio.

Challenges we ran into

Our primary challenge was hardware integration, as this was our first hackathon project involving physical components. We operated on a minimal budget, which meant our components were not high-end. This created numerous issues: the low-quality microphone struggled to pick up voices accurately, which often caused our "Jeff" voice assistant to fail.

Similarly, the basic camera we used wasn't precise enough for the reliable facial recognition we had planned, making that feature difficult to implement. Beyond the component limitations, we faced a steep learning curve in simply interfacing with the Raspberry Pi and getting the microphone, camera, and speaker to all work together with our software.

Accomplishments that we're proud of

We’re proud that we built real-time two-way translation, created a custom driver for our transparent display, fully integrated all the hardware onto a single Raspberry Pi with a 3D-printed face-shield mount, and implemented our wake-word “Jeff Mode,” a Siri-like assistant that provides live guidance and RAG-powered clinical context.

What we learned

We learned a great deal about interfacing with the Raspberry Pi, as this was our first hardware-focused hackathon after primarily building web and mobile apps. We successfully integrated and operationalized the mic, camera, and speaker system, gaining hands-on experience with real-time voice recognition and live video capture for GPT-powered analysis.

What's next for ApexNurse

We want to build a slimmer version where the entire face shield acts as a transparent display with eye-tracking to place information exactly where the user is looking. We also plan to upgrade to a more powerful onboard computer, since the Raspberry Pi can’t support a full on-prem final product. Looking ahead, we hope to run a formal research study to improve the system’s efficiency and explore turning this into a startup that can make a real impact in nursing and healthcare.