Inspiration
A few weeks ago, I was talking to my patient when suddenly his heart stopped. I hit the code blue button and started CPR. Thankfully - with the team, we managed to get his heart beating again. Forty-five minutes later - we found that he had a tension pneumothorax – another life-threatening emergency. The doctor immediately called for a chest tub, the only tool to fix this.
A nurse ran to the supply room to get it, but a minute later, she still hadn't returned. Another nurse was sent to help, but likewise, she hadn't returned back within a minute. Even the doctor went to help them with what should be a pretty straightforward task - finding a critical product in the supply room.
See, the true stress that day wasn't when the patient's heart stopped beating; as a nurse, performing CPR is second nature to me. Instead, it was in that feeling of sheer helplessness when the whole team was just standing there in the patient's room, fully prepared to act but unable to because we couldn't locate the chest tube.
I think about that day a lot, about how we're trained for every emergency, yet how an overlooked flaw in our system compromised our preparation. We had the knowledge and skills needed to act quickly, but the equipment was hidden somewhere in the supply room along with 300 other items.
This wasn't the first time supply delays slowed us down, but that day I knew we had to find a better way.
What it does
ShelfSense is a smart tool designed to help nurses quickly locate items in supply rooms. Just say, “Hey Shelfy,” followed by the item’s name or description. Shelfy responds by announcing the exact location out loud and illuminating lights on the shelves to guide you directly to it. This voice-activated, light-guided system saves precious time, allowing nurses to focus more on patient care and less on searching for supplies.
How we built it
Our solution is built using state-of-the-art artificial intelligence from OpenAI, integrating advanced audio and embedding models. The system first transcribes voice commands into text, then leverages embedding technology to identify the requested item, and finally uses TTS (text-to-speech) to verbally guide users. For light guidance, we employ a Raspberry Pi system running a Flask API server to control shelf lights. The front end is developed in Flutter, providing a seamless user experience.
Challenges we ran into
We faced the most challenges in the hardware of our solution. Initially, we started on our project using an LED strip to guide nurses to the correct location in the supply room. However, we encountered significant obstacles when testing it with the Raspberry Pi. These issues were in externally managed environments that could not be modified. After several attempts of trying to solve the problem using a virtual environment the code ran properly but without activating the lights. From there, we decided to pivot and utilize individual lights instead. This change ensured reliability and functionality in our system.
Accomplishments that we're proud of
Despite the initial setbacks, we are proud of how quickly we managed to integrate a sophisticated software and hardware solution. Our approach has the potential to significantly enhance the efficiency of supply room operations, which ultimately contributes to improved patient care and even saving lives. The development and implementation of our system reflects our team’s dedication and problem-solving abilities.
What we learned
Throughout this project, we gained knowledge about the Raspberry Pi platform. We learned about its capabilities, how to troubleshoot coding issues, and the importance of selecting the right hardware for our solution. Along with this, this experience taught us valuable lessons about adaptability and resilience, emphasizing that challenges can lead to better solutions when approached with an open mind and a willingness to learn.
What's next for ShelfSense
We plan to test Shelf Sense in a Duke Hospital supply room to assess its performance in a real-world setting. With an established partnership, this trial will provide valuable insights into its effectiveness and highlight areas for improvement to enhance efficiency and patient care.
Based on trial feedback, we’ll refine the core functionality, improving both audio and visual guidance for faster item retrieval. Next, we’ll develop flexible setup options to ensure easy integration into various supply rooms, regardless of their organizational methods.
In the long term, with Shelf Sense widely adopted in supply rooms, we’re positioned to transform supply management on a larger scale, introducing features like detailed product databases, inventory tracking, and automated ordering.
Log in or sign up for Devpost to join the conversation.