The medical community and hospital infrastructure were put to the test with the quick and rapid appearance of COVID-19. Hospital efficiency needs to streamlined in order to optimize the delivery of health care to patients who urgently need it. This means expediting as many tests as possible as well as being able to monitor patient status that could indicate detreating health.

Additionally, "by closely monitoring the molecular and immunological data in 326 cases of COVID-19 patients, we suggest that adverse outcome is associated with depletion of [white blood cells]" (Lu, 2020). Thus, a connection between more efficient devices in the hospital and a need for measuring white blood cell count (WBC) is established.

The proposed solution is a medical device that takes an existing pulse oximeter, which already measures SpO₂ and BPM, and attaches a camera to take videos of blood cells in capillaries. This video of blood cells moving in capillaries can be correlated with WBC condition, from which one can extrapolate relative WBC.

What it does

The Meter One device can accomplish and acquire several metrics, such as:

  • Measure blood oxygen saturation
  • Record heart rate and beat per minute (BPM)
  • Relative WBC from WBC condition

Using IR and Red LEDs one is able to mathematically calculate blood oxygen levels and heart rate; however one can also use a camera to non-invasively determine WBC. The signal from the PIN diode and camera is amplified and filtered before being converted from an analog to digital signal using an ADC. These signals are then put through VGG16 LSTM RNN that is able to obtain a relative white blood cell count from the image analyzed. Clinicians have the option of connecting devices via USB or Bluetooth and are able to check relevant data points such as oxygen blood saturation, BPM, and white blood cell count.

Physicians can now get relevant clinical insight from a non-invasive method of data acquisition, this not only increases patient satisfaction but overall health care quality.

How we built it

Hardware Hack

Since access to hardware was extremely limited, even with the assortment of electrical parts I had lying around, I had to rely on using emulation software and work done by others. I had used Altium to build the actual PCB schematic and 3D model. The approach was to take an existing pulse oximeter module and append it with an additional camera sensor.

This hardware hack was pretty theoretical but still serves as a solid proof of concept for some of the capabilities that this medical device can do.

ML Pipeline

We used one of the sample videos of length 60 seconds from the research paper's resources page. We preprocessed the video and enhanced it using ffmpeg followed by cropping the video to separate out the normal and diseased parts from it. We then split the video into 2-second segments in order to increase the number of samples and got about 30 samples for each class. We then used a VGG16 pre-trained on imagenet dataset to extract features from each of the frames in the 2-second video segment and then track these features temporally using an LSTM RNN. We then used the trained model to carry out classification over these video segments at a 70-30 split, 20 epoch, and 4 batch size. We also tested out a VGG16 Bidirectional LSTM wherein we removed the fully connected dense layers from the VGG16 network to obtain high dimensional features from frames and then using a bidirectional LSTM for training and validation. We were able to achieve a prediction accuracy of 93.75% without any overfitting from the VGG16 high dimensional Bidirectional LSTN RNN model. This suggests that this neural network can be effectively used for classification and would provide an accurate fit upon provision of more data. The results can be seen in the attached images.

Flutter Front End App

We utilized Firebase's Cloud Storage back-end and linked it with a Flutter web front-end so physicians can upload device recordings with ease. There are no file-size limits nor amount limits for the recordings. The database is secure, updated in real-time, and the recordings are named in numerical order.

Challenges we ran into

The biggest challenge that we had was creating a good neural network without the actual data from the hardware device. We had decided to use video data from a study that used an optical sensor to get similar video data. We decided that these videos were going to be representative of the actual data acquired from the device. Once we had this, we had to determine what electrical components we needed to add to a pulse oximeter that could gain somewhat similar video data. Ultimately, we were able to create a fairly decent neural network via a creative solution for data acquisition.

Accomplishments that we're proud of

Despite such a lack of hardware and a drought of usable images and data, we were able to complete a hypothetical model of the functional app that includes a user interface and a proper implementation of the VGG16 neural network that allows for the conversion of a digital signal to the interface. Overall an eloquent solution despite all the technological barriers experienced.

What we learned

There was much to be learned regarding the state of hospitalization demands and the chaotic nature of data organization that could lead to suboptimization of patient care and the need to resolve such an issue in such critical times such as the peak of the COVID19 pandemic.

What's next for Meter One

The next step would ideally be to consolidate as many medical devices into the app as possible to allow for a more diverse implementation. In addition, gathering more patient data stored on the cloud-based solution to improve the neural net and hopefully integrate with medical facilities to test solutions and scenarios.

Built With

Share this project: