Nightingale

Exploded View of Wearable
Live Feedback
Diagnostic Results
Render of Wearable

Inspiration

One of the members had to receive multiple surgeries at the knee, head, and jaw, with most requiring additional treatment due to complications. Following further research into academic journals, we discovered that these complications may be mitigated or entirely prevented by monitoring symptoms during sleep. Thus, we ought to resolve this unmet need through the wearable device Nightingale.

What it does

Nightingale is a longitudinal deterioration detection system for patient monitoring post-surgery. Rather than having a patient have to report to the doctor every day or risk not having their care met, Nightingale allows for remote, real-time support through its combined analysis of live sensor inputs and stored memory on the patient and disease through its multi-agent system. Nightingale takes in live information on the patient’s sleep, including things like the number of sleep disturbances. It combines this stream of information with information about the patient’s health trends over the days post-surgery, as well as information mentioned verbally by the patient each day. Using this data, our system reasons through potential escalation and develops a plan that recommends future steps for the patient and caretaker.

How we built it

Hardware

We utilized an Arduino Uno R3 along with a series of analog and digital sensors. A large breadboard was used to simulate and test each sensor individually, which led us to focus on the photoresistor and tilt sensor. A simplified circuit was then transferred on a small breadboard in order to wear comfortably in a wearable device. Following product market research, we settled on an octagonal-shaped chest sensor, which would be able to measure the current parameters accurately and integrate more later down the line (heartbeat, respiratory levels, temperature, etc…). The hub and shield to host the circuit were designed in Fusion 360 and 3D printed for demonstration purposes using the Bambu Lab P1S and PLA.

Software

Our software stack is a full-stack web application with a backend powered by FastAPI and a custom frontend interface for real-time monitoring and visualization of patient and model feedback. We receive live input from the Arduino, which we process and display visually on our webpage.
Our software consists of multiple AI agents powered by Fetch.ai, each which is responsible for a different crucial step in our pipeline. The first 3 (signal, trend, self-report) work simultaneously, allowing for analysis and model feedback on these 3 separate inputs. The results from the models are passed into a chain of models (medical RAG, skeptic, reconciler, clinical brief) that use patient data and published trusted information about the disease to make a decision about whether to escalate or not. These agents work together in sequence to generate a risk score based on reasoning and how severe the issue is, and generate an SBAR report to explain next steps and communicate their findings clearly.

Challenges we ran into

Hardware

Many of the hardware components were already rented out, including the ESP-32, which would have enabled wifi capabilities and easier data transfer, and sensors measuring temperature and acceleration. We replaced the accelerometer with the tilt sensor to conserve the empirical data on sleep movement. Other sensors that were available, such as heart sensor and touch detection, did not provide reliable output due to low quality. Thus, we utilized the accurate tilt sensor and photoresistor to provide our physical data, and resorted to using synthetic data for the missing sensor. We developed a pyserial code to transfer the data from the two available sensors into a usable format for our AI model.

Software

We wanted to ensure our sensors would provide proper and useful medical data while also being practical and not cumbersome to use. In addition, coordinating multiple AI agents while ensuring their predictions and recommendations were based on factual medical data taken from quality sources was a challenge that we needed to spend time discussing to resolve.

Accomplishments that we're proud of

We were able to effectively incorporate hardware and software components into a robust product and smooth user experience that is tailored to different levels of medical expertise. We are also proud to have been able to utilize a vast number of this year’s sponsors (FetchAI, Anthropic, Deepgram, and BrowserBase).

What we learned

We learned how to utilize healthcare related sensors such as heart rate, tilt, photoresistor, and touch and how to analyze the measured data using trained AI models. Most importantly, we learned to think quickly on our feet, switching and substituting different sensors and AI models in order to provide a viable product in 24 hours.

What's next for Nightingale

We aspire to incorporate a heartbeat, respiratory level, accelerometer, and temperature sensor into the device in order to feed empirical data into our AI model. Additionally, utilizing an ESP-32 and custom PCB will enhance the user experience by transferring data through wifi/bluetooth and reducing the size of the hub and shield.