Custom on-prem Object Detection of Medical Devices with RPA

• 

  Object extracted and processed to enable automated extraction of data

• 

  Validating the metrics extracted

• 

  User validating content extracted by Model

• 

  Reviewing the object detected and confidence level

Inspiration

Pushing the boundaries of what RPA is traditionally seen as being capable of in the context of the COVID-19 pandemic.

What it does

Our prototype ingests cell phone pictures of running medical devices (oxymeters or thermometers), automatically identifies the devices, cleans up the resulting image and extract the metrics from the device's screen to feed them into a backend system (or a user for review in case of issues).

How I built it

We built a custom Tensorflow model built on a small dataset of medical devices (approx 300) (Faster-RCNN), custom image processing / clean up routines using OpenCV (custom thresholding). We have packaged this directly into RPA to make it reusable and easy to deploy / run locally.

Challenges I ran into

Training the ML model was a bit challenging but with determination we made it work and documented the process / built tooling to enable our team to quickly train new object detection models and consume them in RPA.

Accomplishments that I'm proud of

Having made it work end to end feels like quite an accomplishment.

What I learned

ML Models can be made very modular and can be used in many contexts within RPA without requiring ML knowledge.

What's next for Custom on-prem Object Detection of Medical Devices with RPA

Refining the model, improving the image processing algorithm.

Built With

• collab
• java
• opencv
• python
• tensorflow