Drone Dr - Remote Triage

Drone Doctor - Rapid Remote Triage
Seeing Triage with new eyes
Overview
Problem
Triage Levels
How it works
Complications
System Overview
Training Watson
Demo
Future Development
Thanks
The Dr. will see you now
Results

Inspiration

We considered a use case of a trainwreck with hazardous materials and the requirement to do rapid remote triage. This drove us to try and find an innovative way to control a drone with limited dexterity that is common with Hazmat and similar anti-exposure suits.

What it does

Drone is launched and controlled using Myo armband interfaced with Flytbase to arrive at disaster site
Photos from the drone are tagged with location metadata and sent from mission computer (raspberry pi) via data links to the IBM Cloud (Watson)
Watson uses Generic and Remote Triage custom classifier to assess the image for initial triage (prone, sitting, standing)
Scene intelligence then pushed to on-scene commander for allocation of resources ## How we built it FlytBase - used Navigation and Telemetry APIs to control and simulate drone
IBM Watson Visual Recognition Engine - Trained custom classifier to recognize prone, erect or sitting persons.
Python - main programming language used to glue everything together.
OpenCV - used to create and display intelligence overlays
Myo Armband - used to control drone functions
Drone - simulated with FlytBase Sim ## Challenges we ran into Some of the challenges that we ran into were getting the Watson IBM to classify our data correctly. None of us had ever used anything like that before so it was quite a challenge. There is a limited dataset of casualty imagery from drones. We also had a bit of difficulty with programming languages and systems not overlapping. We wanted to do most of our development in the Linux environment, but the Myo Armband's SDK only works in the Windows environment. ## Accomplishments that we're proud of Great team photo! Great teamwork! ## What we learned We learned quite a bit. Watson integration was a huge part of this project and, now that we know how to create classifiers, we can do all kinds of things with that. We also learned all about Quaternions and how to classify rotational data with the Myo Armband. Learning about how emergency response works was also really interesting ## What's next for Drone Dr - Remote Triage
Modify Marine Maker (MD5) Nibbler 2 Drone with code to perform rapid remote triage
Implement automatic scene segmentation of captured images
Update and expand Remote Triage custom classifier
Assess purposeful movement from video (SALT Step 1 - Assess 2nd)
Perform Individual Assessment (SALT Step 2) with IR camera to ascertain respiration rates and major hemorrhages
Report victim locations with triage assessment on map
Enable drone swarm control

Built With

drone
flytbase
ibm-watson
myo
opencv
python

Submitted to

XBuild 2018
- Winner Third Prize

Created by

I worked on the Myo Armband and FlytBase Drone Simulation side of things.

I was able to import the data from a Myo Armband and translate that raw data into an actual usable yaw, pitch, and roll. Using FlytSim was actually a lot less daunting than it seemed at first also. I used their web based drone simulator and web API to send it commands based on the gyro data from the Armband.

I created different python scripts to run based on different functionality. So, there was one to takeoff/land, one to grab lat/lon data, etc.

Tyler Gragg
For this project, I have worked on the integration of Watson cloud computing AI technology in our operational scenario. I have also created annotated image rendering with OpenCV to report the intelligence results to the ground station.

david-beaudette
I trained a custom Watson AI image classifier to detect and classify individual images for rapid remote triage. I developed the story line, built the slide deck and generated all logos and images. I also provided inspirational leadership and adult supervision.

Josh Dittmar
Michael Henley