Sauron

Inspiration

The creation of Sauron was inspired by the recently increased rate and severity of humanitarian crises around the world. From environmental emergencies to political instability to infectious disease outbreaks, now more than ever the public needs (and deserves) accessible information about current events. Sauron is a data pipeline designed to distribute information during humanitarian crises by combining the power of cloud-based AI with the reach and scale of publicly available assets (traffic cameras, satellites, etc). By detecting and mapping hazards in real time, Sauron empowers anyone with an internet connection situational awareness when they need it the most.

What it does

Sauron's operation cycle begins with the procurement of images from publicly available assets. These can be images taken by a satellite, or video feeds from local cameras (which can be sampled by frame). These images are then sent through machine learning based computer vision algorithm running on a cloud platform to be matched with an incident type (assuming an incident is detected). These images are then sent to Sauron's main webpage, where they are projected onto a map of the crisis region and sent into a feed displaying all nearby hazards. By going to Sauron.com, any user can now see icons indicating events on a map of their local area as they are occurring.

How we built it

At HackPSU, our team's goal was to prove Sauron's main concept, which has 4 essential components:
1) Collecting images from publicly available sources
2) Sending these images to a computer vision model that can detect hazards
3) Displaying these events on a map available to anyone with an internet connection
4) Doing this all in real time.
To prove the viability of this concept, a minimum viable product was created. This minimal viable product is hosted on the Google Cloud Platform (GCP) with Linux VM instances. The data is downloaded from YouTube traffic camera livestreams, aggregated in a GCP-hosted MySQL database, and visualized using Grafana, which natively integrates to the system. The anomaly detection algorithm utilizes OpenCV and is written in Python. To demonstrate the fact that this algorithm is analyzing real-time data, white squares are randomly overlaid on the images as they are sent to the algorithm. OpenCV then detects these squares and signals to the database that an event has occurred. Lastly, to demonstrate the scalability of Sauron, a web app was developed using Flask so that bystanders to events can upload images of them with the appropriate location and timestamp.

Challenges we ran into

The majority of our challenges surrounded the integration of various system components with one another. Because the data pipeline has so many different systems (GCP, MySQL, Grafana, Flask web apps, etc) ensuring they were developed compatibly was crucial. This was made more difficult by the varying level of Python experience and Linux OS use in our team. This challenge was overcome by open communication between team members and repeatedly diagramming the inputs and outputs of each system.

Accomplishments that we're proud of

Our proudest accomplishment in this challenge was the development and connection of the entire data pipeline as a proof of concept. Sauron, even in this very early stage, takes an image from the designated video source, sends it through a computer vision algorithm, and outputs the location of detected incidents to a public webpage all in real time, autonomously. While some system components require further iteration (implementation of machine learning, data collection from new sources, etc), the underlying structure that will support these changes is developed and actually running with only 24hrs of development, which in our opinion is a huge accomplishment.

What we learned

All of our team members gained valuable experience in the development of complex software solutions by participating in HackPSU. For context, two of our members are experienced programmers in EECS, while the other three are beginner to intermediate programmers in aerospace engineering. None of our members have ever implemented project in GCP. Furthermore, our EECS members had limited exposure to AI and our other members had none at all. Lastly, with the help of our members with more programming experience, the remaining members acquired the following skills:
- Python
    -Image Processing with MoviePy & OpenCV
    -App development with Flask
- Linux OS use in a virtual machine
- JavaScript & HTML
- MySQL

What's next for Sauron!

The continuation of Sauron post-HackPSU will have two main pillars. 
1) The implementation and training of machine learning models into the computer vision software. This will allow us to detect actual events in unaltered footage, and to classify these events by type (natural disaster, armed conflict, etc).
2) Expansion of Sauron's access to image sources. Short term developments include allowing users to stream phone video to Sauron via app, and directly accessing CCTV cameras not streamed to YouTube. Longer term improvements would include the integration of satellite imagery and even local unmanned aerial vehicles.