Hold Steady

Inspiration

Hold Steady was inspired by the need to enhance surgical care, boost attention, and reduce the amount of mistakes made on the surgical table. Depending on their field, surgeons have up to four daily surgeries that could last for hours each. Due to this high-stress environment, even the most dedicated surgeons can tire, and even the slightest misstep can cause drastic consequences. These surgeons, however, may be so enraptured with their work that they might not even notice their fatigue. Hospitals try to combat this by using buddy systems to modify work schedules; however, a non-biased approach is critical during surgeries to ensure patient safety. We aim to reduce the number of errors these surgeons make through computer vision.

What it does

Hold Steady comes with an app and two wristbands that will be used together to reduce errors during the surgery. Hold Steady is an app that can track, report, and record how shaky the surgeon's hands are during the surgery. Our technology uses computer vision software and an inertia sensor embedded in the wristband. Computer vision, the primary system, will track the highlighted red box to check if the surgeon's wrists are shaking. Then, it compares this data to the secondary system, which is the inertia sensor, to determine whether the surgeon's hands are shaking. By having two systems, we can ensure that the most accurate report is given to the doctor and nurse. Given the circumstances of a surgery, we do not want to falsely alert the doctor, which can cause unnecessary stress.

In the Hold Steady system, the doctor wears the main wristband, while the nurse wears the buddy wristband. If the software determines that the surgeon's hands have become too shaky, the buddy wristband will receive a continuous vibration until the surgeon's hands stabilize. Using their own judgment, the nurse can then alert the doctor to rest if necessary. This collaborative setup ensures that the doctor isn’t bothered during the procedure, and is only notified when deemed necessary. In this joint effort, both the surgeon and the nurse are actively involved in maintaining the stability of the surgical procedure.

Hold Steady offers post-surgery benefits as well. Through the app, Surgeons can access their previous surgeries and track the points at which their hands shook above the threshold during the surgery. A graph of “shakiness” versus time will be displayed for each surgery, with a horizontal line signaling the threshold. Each period of time above the threshold will be noted down as a shake, and the total number of shakes will be reported. The average length and total duration of shaking will also be reported. This data can then be used to create more effective work schedules for these surgeons, potentially reducing the risk of errors on the surgery table.

How we built it

Hold Steady consists of two bracelets and an app. We first designed the bracelets using both rubber and metal. We have an elastic rubber band that fits all in one size. We attached a metal clasp on this elastic rubber band containing either a motion sensor or a vibration motor.

There are two distinct bracelets: the main and the buddy. The blue/red bracelet will be the main bracelet, i.e., the surgeon's bracelet, as it contains the motion sensor, while The camera can capture the bright red metal on the main bracelet and detect its movements. It can also detect the surgeon's movement through the motion sensor embedded into the metal. The pink/white bracelet will be the buddy bracelet, i.e., the nurse's bracelet, as it contains the vibration sensor. This way, the buddy bracelet can be notified without disrupting the surgeon when the main bracelet detects motion.

Challenges we ran into

Computer Vision: One of the most considerable challenges was finding ways to monitor the red band due to different light conditions and angles. Due to how light reflects off the band, it may cause a misread from the software side. Another problem we faced was detecting the red rectangle from different angles. To counteract these problems, we train the model with different vividness and orientations to ensure we can accurately predict where the box is at any given time.

User Error: Another challenge we faced was distinguishing between the surgeon's normal instrument handling and genuine handshaking. To overcome this, we developed a highly precise algorithm that can detect the presence and intensity of handshaking, ensuring the system only alerts the surgeon when necessary.

Patient Safety / Design: Surgeons need to concentrate the most when focusing on the surgery. If a phone rings or something on their person is even slightly bumped, they can cause problems from internal bleeding to destroying vital organs. We struggled to find a solution that can notify the surgeon if they are shaking, so we decided that human interaction is the best in this case, as a person who is not operating on the body can notify the surgeon at their discretion.

Accomplishments that we're proud of

Seamless Integration of Computer Vision: We successfully developed a system that uses computer vision to track and report hand shakiness accurately in real-time. This technology enables us to minimize false positives and ensure surgeons are only alerted when necessary.

Non-Intrusive Alert System: One of our biggest wins was designing a notification system that doesn't interrupt the surgeon's concentration. The buddy bracelet system allows for discreet and controlled alerts, giving the nurse the final say on whether to notify the surgeon. This reduces unnecessary distractions during critical moments.

Accurate Motion Detection: After extensive training and testing of the model under different lighting and angle conditions, we overcame challenges in reliably tracking wrist movements. The system can now detect subtle shakiness while distinguishing it from routine hand movements.

Data-Driven Post-Surgery Insights: We created a comprehensive post-surgery tracking feature that allows surgeons to review their performance. The app generates detailed reports on shaking episodes, providing valuable insights to help surgeons optimize their work schedules and reduce fatigue-induced errors in future surgeries.

Human-Centered Design: We are proud of our solution, which prioritizes patient safety and surgeon comfort. By ensuring that the system is supportive and not intrusive, HoldSteady enables surgeons to perform at their best while receiving timely assistance when needed.

Functional and Ergonomic Bracelet Design: We designed two distinct, functional bracelets that are comfortable and durable. These bracelets ensure that the motion and vibration sensors work seamlessly during lengthy surgical procedures without causing discomfort to the users.

What we learned

In this project, we used YOLO (You Only Look Once) for object detection. Due to its speed and accuracy in real-time applications, it is particularly effective for detecting small objects in motion. The algorithm is designed to process images in a single pass, which makes it ideal for detecting objects in fast-moving environments. In addition, YOLO's ability to identify distinct features like color and texture helps detect the red metal if it contrasts with the band.

As we built the project, we learned how to set up YOLO architecture and conduct successful training sessions. We created a valid dataset and generated meaningful labels that helped us train a model with high accuracy. We also learned how to run inference sessions on Google CoLab. During the process, we used Pytorch and other essential libraries to work with machine learning and big data.

What's next for Hold Steady

Customization Thresholds: We understand that different medical fields have various levels of leniency with movement. We will allow the hospital to set these thresholds as they see fit. Note that we recommend that this app should not be a metric for hospitals to track surgeons, as Hold Steady aims to reduce errors within the surgery and not worry the surgeons during surgery.

User / Regional Research: We will contact local healthcare providers and provide a sample of the product, specifically ER and other high-stress environments within the medical field. We will then gather and implement feedback to help serve our community.

Online and In-person Training: Within hospice care, we plan on ensuring that both the doctor and the nurse understand the significant benefits and usage of Hold Steady, so that they may harness the true potential of our product and our main goal: ensuring patient safety.