OH, OH, Au Secour

Inspiration

The lack of nurses in hospitals is a growing problem in Quebec's health system. In 2024, we hit a record low for the number of new nurses in the province (Radio Canada, 2024, https://ici.radio-canada.ca/nouvelle/2041039/infirmieres-examen-ordre-dube-lebel). This is leading to growing wait times in hospitals, which in turn leads to patients suffering longer in terrible conditions. This is why we have come up with a patient monitoring robot, which could replace many nurses in the upcoming years, taking a significant load off our health system.

What it does

Our prototype can be put into a nursing home, a hospital, or anywhere else where there are people at risk of fainting, falling or injuring themselves like construction sites. There is a ground station with a web ui which the robot can connect to, giving the operator a live video and audio feed of a camera placed on the robot. In idle mode, the robot will periodically take pictures of its surroundings and prompt google Gemini to determine whether or not the people in that picture are injured. If Gemini returns that a person is falling or injured, the ground station will set off an alarm for the operator to see, and the robot will switch to manual control mode, giving the operator full control of the robot to navigate to the injured person via the camera on the robot and the operator keyboard and talk to them via the speaker and microphone on the robot to assist them. The operator can at all times switch the robot to manual mode at his discretion to watch over specific areas.

How we built it

Our prototype was built using a styrofoam base, a rasberry pi, servo motors, a webcam and a bluetooth speaker. We programmed our backend in python, our frontend in raw css / js and the embedded code for the pi in python as well. We chose python and javascript for their ease of development, as speed is key in projects that are heavily time constrained like hackathons. The electronics (servo motors, buck converter, webcam, bluetooth speaker, rasberry pi) were picked arbitrarily from the electronics we had brought, as we had absolutely no plans and brought along a ton of assorted electronics.

Challenges we ran into

The audio streaming was a big challenge. There were many synchronization issues that popped up as the project went along, and as this was only our first or second time programming on a rasberry pi, we were not very familiar with the environment, the libraries and the python programming language. This made getting a synchronized clear audio stream very difficult, as we had to set alot of the audio streaming parameters ourselves for our devices to stream properly. The assembly of the physical robot was quite a challenge as well, as we realized the day of the hackathon that we could not use most of our motors for various reasons like movement inaccuracy or power constraints. We ended up using one servo motor and one DC motor for the movement, which were incredibly challenging to get to work together.

Accomplishments that we're proud of

We are incredibly proud of having brought a physical robot to life in such a short time span, and above all that it works ! Alot of these technologies (ai API requests, servo / DC motor coordination, webrtc) were very new to us and we are so proud of everything we've learnt !

What we learned

We learnt how to use an API to request prompts to an LLM for image recognition, which made the recognition of injured people possible for our project, how to program embedded software for a rasberry pi using python to enable us to synchronize audio streaming between devices, how to use webrtc for real time streaming and how to synchronize DC and Servo motors.