SeiSure

Privacy-first seizure monitoring using computer vision and physiological signals.

Comment

Inspiration

Epilepsy can result in dangerous and long seizures, resulting in a much higher risk of premature death. However, current solutions for this include continuous caregiver supervision, specialized service animals that can be extremely expensive, or clinical-grade monitoring such as EEG systems that are difficult to use continuously in everyday life. We aimed to create a more accessible solution that would help discover patterns that occur before a seizure, quickly identify when seizures happen, alert caretakers when it does, and keep track of important metrics to further understand how and why they happen.

What it does

SeiSure is a multimodal seizure monitoring prototype that combines heart rate data from Apple Health with computer-vision based movement analysis. A secondary IOS app included reads heart rate measurements from HealthKit and sends them to the SeiSure dashboard. The browser uses computer vision to track body movement from a live camera feed. This way, just an erratic arm movement such as stretching would not trigger a false positive, and a certain amount of both signals over a few seconds is needed to detect a seizure. When detected, SeiSure will alert the caretaker or allow the user to hit a "false alarm" button.

Additionally, SeiSure is aimed at collecting data for research purposes. It records info such as which limbs move more, how much the increase in heart rate is between a pre-seizure and seizure state, and other valuable information that could later be used to improve the detection.

Since adults may feel uncomfortable giving their caretaker 24/7 access to watch them, we also made a privacy mode that skeletonizes the video so our computer vision models and caretakers can watch and keep an eye on the patient without seeing details about their appearance.

There are additionally more customizations that patients can make, such as creating a profile, writing in caretaker, medication, and other information, and adjusting the thresholds for seizure detection by personal preference.

How we built it

SeiSure is made of a React/Vite web application for live monitoring and analytics, and a Swift iOS HealthKit application. The visual component uses MediaPipe Tasks Vision and Pose Landmarker models to get body landmarks (like shoulder, right arm, etc) from the video, and we track changes in these landmarks to estimate whether movement is getting faster. Additionally, the movements of specific limbs is documented to see whether certain limbs are more prone or reactive. That way, we get an overall motion score and scores for each limb.

For heart rate monitoring, a Swift iOS application using HealthKit reads the latest heart-rate samples available from Apple Health and transmits them over the local network to our application with a HTTP API over the local network .The web application was built with React and Vite, with local API endpoints connecting the browser and HealthKit application. The browser polls the heart-rate endpoint and the MediaPipe processes the camera data.

Challenges we ran into

One of our biggest challenges was utilizing the heart rate data from Apple HealthKit, for several reasons. For instance, most of the processing happened on the computer, but since apple HealthKit requires a phone, we had to create an iOS application to effectively send the data. We had to build a bridge that could transfer this data, and issues such as lag between the laptop and the phone and Apple Watches not constantly sampling heart rate data, it made this a less reliable data stream than we would have liked.

Additionally, the computer vision also has difficulties, as it often glitches, misunderstands positions in space, and can contribute to a higher number of false positives.

Accomplishments that we're proud of

We are proud that we were able to build a companion application and actually utilize the apple health data, as utilizing live apple health data is always difficult to work with. Our application is an actual full stack working application rather than being hard coded to only survive the demonstration. We are especially proud of the privacy mode, as privacy is not a factor that other projects we have researched have typically considered in the past, setting us apart. We strongly believe that all patients deserve privacy and the ability to feel comfortable and not surveilled while also feeling safe.

What we learned

Heart rate from Apple Watches is unreliable, for future work we plan on looking into better wearables including better heart monitors as well as wearable EEGs. We also learned that adding another data source changes what we predict would look like a "real" seizure significantly. For example, while we were testing our product, we'd be flailing for a few minutes but still fail to trigger an alert because our heart rates were still too low to become suspicious.

We also learned that forming a threshold with multiple data sources is difficult, as there is not a significant body of research that tells us definitely that a specific heart rate and amount of movement is guaranteed to be a seizure, so these numbers have to be derived by trial and experiment.

What's next for SeiSure

Our next step would be to collect data from realistic scenarios and see if we could fine-tune the thresholds to ensure they best matched the patients. Additionally, either swapping the Apple Watch for a different source of physiological data or simple adding more sources on would help us further improve our technology to have a better accuracy.

An additional next step would be user testing and validation. Due to no one on our team having epilepsy (and epilepsy typically not being an "on demand" disorder), it may be difficult to collect a significant volume of data that supports the strength of our approach, but we hope that some testing will reveal designs that we could improve on.

Longer term, we hope SeiSure can help patients, caregivers, and researchers better understand when seizure-related events occur while collecting as little identifiable information as possible.

Built With

Share this project:

Updates

Submission history