RescueMate - A Novel AI Assistant for Emergency Situations

The Problem

We address a gap in emergency medical response time for remote and underserviced locations. Currently, it takes 7 minutes for emergency medical services (EMS) to reach the scene, with the time in rural areas doubling to 14 minutes. For critical medical emergencies, such as cardiac arrest, every minute the chances of survival decrease by 10% making immediate intervention necessary. This is where the most crucial element to save lives comes in: the bystander. In fact, the intervention of bystanders can triple the chances of survival in instances of cardiac arrest.

Here is where another issue arises. Only 6 out of 10 people feel comfortable even attempting to perform CPR for someone in cardiac arrest. This stems from only 3.5% of people in the United States being trained in first-aid procedures such as cardiac arrest, and an irrational fear the bystander will inflict further damage to the victim.

The Solution

Our novel AR and AI-based pipeline equips bystanders with the knowledge and visual guidance to perform life-saving procedures with confidence and precision. Using RescueMate, the device is able to detect a medical emergency and guides the bystander on how to care for the victim until emergency medical services arrive on the scene in order to give the victim the best chances of survival. RescueMate acts as a medical professional in the room that you can talk to, with the goal of guiding you through the process of delivering life-saving treatment. Not only that, but with the power of Google's ARCore, RescueMate can add immersive 3D human models that perform CPR on the injured person, showing the user exactly how to perform the procedure.

Our app offers two modes: practice and emergency.

Practice mode is designed to make medical training for emergencies more accessible and environmentally friendly. Users are provided with a multitude of training options, each tailored using ARCore models that leverage the depth API and environmental understanding. Each training option includes a personal Gemini Assistant that provides directions and feedback, constantly updating to adapt to the environment. This feature allows users to avoid purchasing expensive and non-recyclable medical practice kits.

In emergency mode, Rescue Mate utilizes the Gemini API to diagnose conditions based on symptoms described by the user. The Gemini API then provides guided instructions in real time, creating immersive ARCore steps that include video nodes, medical models, and more. The ML Kit Pose Detection API is used to locate body pose landmarks and accurately place these 3D models. Once first responders arrive, the Gemini API provides a detailed summary of the emergency, including important information such as patient symptoms and elapsed time, helping first responders and reducing the bystander’s stress.

How we built it

We used Google's Gemini API to create a multimodal AI voice assistant that could see the world while also responding to questions by the user. Gemini API was also used to create guided instructions in real time that were used in conjunction with Google ARCore to add immersive video nodes, medical models, and more. In order to place these 3D models in the right places, ML Kit Pose Detection API was used to locate body pose landmarks.

Built With

• arcore
• geminiapi
• mlkit