Cherub

A behind-the-scenes glimpse of the Firestore database.
Website Landing Screen
Website Empty Search Screen
Website Contact Screen
Website Results
Website Search Specific Results
Door Selection
Computer Vision in Action 1
Computer Vision in Action 2

Inspiration

These past few weeks, we've all had the experience of going to the store only to have to cue up outside because the store was at capacity. As we speak, more businesses are opening, but many are restricting the number of customers that can be in the building at one time due to the pandemic. For one, it's hard to track the number of customers in large buildings, such as malls. Moreover, even when the number is being tracked, it isn't always available to consumers unless they physically go to the store. Even when it is, the crowd information for many different stores is rarely aggregated in one place to allow customers to make an educated decision about where/when it is safest to shop.

What it does

Using cameras and CCTV in public places, CHERUB provides real-time information about crowd levels to users. As an added bonus, CHERUB also provides users with information about mask restrictions. The user can search for locations by a specific category and their location and CHERUB will provide them with a list of places sorted by proximity. This information is all available at https://cherub-9df2c.web.app/#.

How we built it

The project involves the interaction of a javascript website front end with a python backend and Google Firestore to communicate between them. The python script takes continuous video footage and uses a detection-tracking algorithm to find people. Pedestrians are found using OpenCV with a pre-trained detection algorithm and are tracked with DLib to reduce computation time. If a person crosses the border of a door, they will either be counted as entering or leaving the location based on where they walk. From there, if the number of people changes by a significant amount (e.g. by 5), the Firestore database will be updated. Whenever the database updates, our Cherub website updates its information to be presentable. On the website end, when the user searches by location and category (e.g. entertainment, food, etc.), a list of applicable locations that have signed up with us will appear to the user. Additionally, to make it user friendly for the store owner, an additional python script was written to allow them to easily set up bounding boxes for entrances.

In layman's terms: A script run on an onsite device continuously analyzes video footage to determine when people enter and exit the location. A live count of the number of people in the place is stored in the cloud. When a user accesses the CHERUB website, the website queries the database for the type of place the user is looking for (entertainment, restaurant, grocery store, etc) and displays those that are geographically closest to the user.

Challenges we ran into

On the database end, integrating Firebase with python and connecting it with Firestore was quite challenging. For the computer vision algorithm itself, tracking where people move and associating movement with entering and exiting a building was somewhat difficult. Determining geographical points based on zip code was also somewhat tricky. As in these cases, if one aspect took longer than expected, a major challenge was developing a way for everyone to keep working even if the parts were interdependent on each other. As for the setup itself, coordinating ourselves without any physical communication was somewhat difficult, as was hyping each other up even when we weren't able to high five each other in person.

Accomplishments that we're proud of

We've had quite a few great moments of victoriously jumping up and down behind our screens, and shouting over each other in the Zoom call. For one, when we first got the Computer Vision algorithm to read in a sample YouTube video we found and count the number of people in and out. Another great moment was bridging the website to the firebase to the computer vision. We were working in four different houses and as many different languages, but we managed to tie everybody's piece together. Completing the user-end website was also exciting because it meant that if we set up our code at a store, via a device like the CHERUB Box described below, it can immediately start benefiting the community. We found a problem in our lives and communities, and we developed a solution that actually fixes it! We're very proud of that.

What we learned

On the computer vision side, we learned how to track and count the number of people entering/exiting an area. In data storage, we learned how to use Google Firestore and host a website on Firebase. On the JS side, we learned how to debug as a team via video call. One of the main learning experiences was how to manage a multilingual project (Major components in JS and Python), and quickly design an overarching structure with everyone working on their strong points.

What's next for Cherub

How it will impact the world

CHERUB can be used in an almost unlimited number of situations. First, it can help promote preemptive social distancing in various locations including stores, malls, theaters, restaurants, and more. The stores simply install CHERUB to run on their CCTV footage. Individuals can access the CHERUB website to determine which locations have low traffic and are safe. However, that's just the tip of the iceberg.

Stores and offices can use CHERUB locally, without sending the information out to the public, to determine high traffic times of day, when social distancing policies should be more rigorously enforced. The CHERUB Box, a black box installation of CHERUB, can be set up outside small spaces such as meeting rooms and bathrooms to provide information about the crowd level inside. This could be accomplished via a small light on the box that would blink red if the room was at capacity.

CHERUB Box can also be set up locally at schools to count the number of students in and out, so that in the event of an emergency evacuation, teachers can quickly determine whether every student has left the building.

Additionally, CHERUB can be applied to manage protests. When many people are moving in and out of an area at the same time, it can be difficult to determine the general trend. CHERUB does the counting for you. It can determine when there are too many people in an area to maintain safe social distance so that the people in the area are aware that it's best to spread out.

TL;DR: Amongst other uses, Project CHERUB promotes the "new normal" by allowing people to plan for social distancing and safe public outings. Visit us at https://cherub-9df2c.web.app/#.

Improvements to the algorithm

The main improvements to the algorithm would be to make it a more user-friendly experience for the store owners. To do this, an intuitive and easy set-up script can be written to walk them through the initial steps. If CHERUB becomes easy to install and get started with, it would promote usage which would ultimately help the community.

The detection of when a person enters a building can also be changed and improved. Our current model only checks whether a person is inside the bounding box of a doorway but this can be problematic if the camera is placed in front of a clear window that is not the entrance (since people inside could be mistakenly counted as leaving the store). Solutions can be to use projective geometry techniques such as the cross-ratio to better determine distances or use cameras from multiple angles to achieve the same purpose.

The pre-trained model used for CHERUB can also detect much more than people. Because of this, it can have applications in other locations. In parking garages, CHERUB can detect the number of cars in or out, letting the user know which garages are full, and which have open space. On a farm, CHERUB can be used to quite literally count sheep (or any other livestock), so that farmers can ensure that all of their animals are safely in for the night.

The CHERUB Box

While some places such as stores currently have live video footage of their entrances and exits, CHERUB doesn't need to be limited to these locations. A small black box computer containing the CHERUB software and linked to an onboard camera can allow CHERUB to be easily set-up and operated anywhere in the world. We aim to partner with the commercial sector and the public to make CHERUB available everywhere where running totals of people in an area could prove useful.

In the era of "new normal," the world needs a way to easily incorporate social distancing into their lives. CHERUB is here to help. And even after COVID-19 is defeated, CHERUB will be here to continue serving the world.