sentrysight

Overview

Transportation systems such as airports and airlines are perhaps the largest problem area when it comes to controlling and preventing the spread of Covid-19. It is here where people from different geographical places and cultures collide in close proximity that make these places hotspots for the spreading diseases such as COVID-19. Besides following recommendations provided by the CDC on how to best prevent the spread of the disease, there is very little technology that is being implemented in the United States to thwart the circulation of the disease through travelers. Currently being tested are thermal cameras but they have several downsides such as travelers may not be showing symptoms or may be taking medication to suppress symptoms. Essentially, trying to individualize the method of detection is both inaccurate and time consuming. The solution is to not detect the individuals with the virus but to provide effective prevention recommendations to travelers and collect and use data on where travelers are following the guidelines. Our product uses existing cameras and machine learning models in these transportation systems to not only determine if travelers are socially distancing six feet apart but also identify if they are wearing masks. From the data collected, both the travelers and airport authorities can access a dashboard to see how people in certain areas are following the guidelines. This benefits everyone since the airport can identify and better regulate problem areas and travelers will be reminded to follow guidelines better and choose to avoid problematic areas.

Team Members

Brandon Wood (Project Manager & Software Engineer, Senior Computer Science @ Purdue)

As the project manager, Brandon set up a gantt chart based spreadsheet and scheduled daily standups to keep track of progress. He also worked on the frontend and the database for the project.

Taraka Kantipudi (Software & Data Engineer, Graduate Student Applied Data Science @ IUPUI)

Taraka worked on the machine learning model for detecting mask wearing and social distancing behavior from the video feed input. He also worked on deploying the model in Google cloud platform for continuous running of the model.

Karen Bonilla (Software Engineer & Designer, Junior Informatics @ IUPUI)

Karen worked on the map display portion of the project. Karen also designed the initial prototype and improved the design of our product using CSS and other methods.

Jacob Hao (Business Development, Senior Information Technology @ Indiana State)

Jacob worked on developing the customer survey and discovery process. He also helped create different tables that better help us identify our users and their needs and desires.

Skyler Ferguson (Business Development, Senior Mechanical Engineering @ Rose-Hulman)

Skyler completed work on the Environmental Analysis, Customer Profile, Value Proposition Canvas and the Business Model Canvas.

How did you decide on this customer segment, problem, and solution?

Initially our team was thinking that we wanted to create an improvement for a single mode of transportation, but then we thought why not take it further? We can improve multiple forms of transportation by creating something that can be used anywhere. We know social distancing is very important along with wearing a mask, so how can we encourage those behaviors? We knew that people could feel more comfortable and businesses could operate in a safer manner if they had real time data on their facilities. We decided to use machine learning integrated into camera feeds to give data on whether people are social distancing or wearing masks. This solution is cheap for anyone to implement as most already have security cameras. In addition, airports already have displays for showing flight arrival/departure times that can be used to also display mask and distancing information. We found there were not many products like this available on the market, so we knew that this is a hole that could be covered. We even put a survey out to the public that allowed us to get feedback directly from the people that this could affect. During this time we learned a lot about working in a team under different constraints, and how we could still work effectively despite that. Public transportation can be scary at a time like this, so making people even be a little safer is the number one concern.

How did your team build and iterate on the solution?

Our team started by branching into groups organized by our technical strengths. We initially tried using github projects to manage workflows, but it did not fit our need to organize both code-based and non-code based work. Our final solution was to create a spreadsheet tracking system inspired by gantt charts that allowed everyone to see all data on workflows quickly and easily. Prototyping started slow due to the difficulty of hard coding our solution effectively. Despite this we were able to use a wireframe model and documentation to assemble our ideas. We decided to create a webapp using React and use Google Cloud as our backend to provide easily accessible and scalable architecture. We used a bottom up approach that allowed us to quickly create key workflows but created later compatibility issues when combining those workflows. However, we were able to overcome these issues by modifying our database schema. Based on user feedback, some small changes to data presentation were made near the end of the project.

Key Metrics

50+ Survey responses

75% of respondents would use our product

Technical Architecture

alt text

Key Tools, Libraries, and Frameworks

Firebase Real time database : We chose Firebase as our database provider because of its ease of use and integration with Google services used in other parts of our project, mainly the Google Cloud Platform. Firebase allows us to store and retrieve data easily in real time from anywhere as our dashboard requires. Additionally, it is the provider where our team has the most experience.

Firebase Hosting : With our database already implemented in firebase, adding hosting via the same provider made sense and allowed us to quickly spin up prototypes to share.

React : We chose React as our frontend framework mainly because of its focus on real-time data, which is central to our solution implementation. We also had more previous experience working with React than other frameworks.

Python : We used python as our core programming language to perform all the operations with regard to machine learning, object detection and identifying the distances between people. Python provides flexibility and an easy to learn language making life easier. Jupyter notebooks make it easy to run small chunks of code with excellent user interfaces during the development phase while python scripts would be more appropriate for deploying code into production.

YOLO : You Only Look Once is one of the fastest object detection algorithms available. It has 80 different classes of objects including humans, vehicles and different other objects. We used YOLO version 3 algorithm for detecting humans which is the key part of our project.

Google Cloud Platform : We used Google Cloud Platform capabilities to make our life easier. GCP features the machine learning model Auto ML Vision which allows for automatically developing a machine learning model. We supplied the labeled data, which we got from kaggle. Using AutoML Vision, we trained the model with 5600 labelled images for people with and without masks. We then used GCP for creating Virtual machines to run our model in a production environment. GCP also provides storage buckets which we used for storing the video footages.

OpenCV : We used OpenCV, a realtime computer vision library used for analyzing the video and image data. We used openCV to process the video input and send the data to the machine learning models. The library is used for writing the video output as well.

Shell Scripting : We used shell scripting to automate the process of running the script and developing a data pipeline making the data transfer and providing required access to the file easier.

If you had another 5 weeks to work on this, what would you do next?

With more time to work on this we would customize our product for more scenarios instead of a one size fits all solution. While right now our product is great for businesses it could do more with the general public. We would also continue working on the technical aspect by making our application mobile friendly and further improving the layout of the site. We would also add more standard features such as authentication, settings selections, and support for non-live streams. Finally, we would reach out to more businesses and get feedback on changes or features we would need to add via a beta test.