Cozy Koalas

Dashboard Home Page
User Page
Labels on images
Example batch run of YOLO model

Inspiration

Positive cases are spreading really quickly on campus, and many students decide against going to dormitories because of ineffectively mask enforcements. We hope to make these places safer so college students can come back to university sooner and enjoy their precious student-life experience more!

What it does

Cozy Koalas allows for the identification of people as well as whether or not they’re currently wearing a mask, using YOLOv5. It would allow dormitories to monitor statistics such as number of people with/without mask throughout the weeks. Another feature of our application is our infrared sensor which detects the temperature of an individual (https://ieeexplore.ieee.org/document/9530864) . If that person’s temperature is abnormally high and may have a fever, a notification is sent using Twilio so that they are aware of this and will take action to reduce risk.

How we built it

Machine Learning: We use images from a camera feed to detect a) whenever a person comes into the frame, and b) whether that person is either:

Not wearing a mask
Wearing a mask incorrectly
Wearing a mask correctly

We use YOLOv5 (You Only Look Once) model, a real-time object detection model based on convolutional neural networks (https://arxiv.org/pdf/2102.05402.pdf) and incorporated a Python script to help label our data.

We ran multiple iterations through YOLO to improve our model and labelling. Initially, we only had 2 labels: mask or no_mask. Howeverk, this was ultimately problematic as our model was unable to detect when someone wore there mask incorrectly (e.g. doesn’t cover nose). That’s why after multiple iterations, we added another label using a Python script. While that was our main change, our multiple iterations helped us balance our data and tune our hyperparameters, leading to a greater accuracy. This backend is connected to a server and a database via Google Cloud’s Firebase for the moment.

Front End: The front end fetches its information from the Google Cloud’s Firebase and displays it in an interactive dashboard. The dashboard and its following pages were done using Material UI, a front-end library in React. Furthermore, a number of other libraries or tools were used to help sort the data/ beautify the application, such as lodash, iconify, ant-design, faker, etc.

Challenges we ran into

We struggled on really understanding the YOLO model and convolutional networks before implementing it. We initially tried to implement parts of it without understanding this nor torch as this was our first time working with it. However, to actually improve our model, we really needed to understand the parts to change. We also struggled immensely on connecting the two parts of our projects together. In the front end, we ran into multiple typeerrors that had to do with states and usages of props.

Accomplishments that we're proud of

We’re proud of how we worked as a team and leveraged our different specialities and managed to create a working product together once we stitched different parts of our project together.

The computer vision and machine learning modules and libraries are state-of-the-art and very much used in current technologies. Our model is even able to differentiate very corner case situations, such as when one has their hand to cover their face instead of a real mask or when the mask is worn incorrectly. The dashboard also turned out to be simple, clean and elegant and reflected what we initially went for.

What we learned

ML models are easy to use yet hard at the same time. Documentation and APIs are useful, but a large part of the understanding is understanding your data and what changes need to be made to improve your model. Data visualisation and metrics were very helpful for this part! Another great thing was the potential and opportunity which comes from already pre-existing and labeling datasets. In our project, Roboflow and Kaggle proved to be hugely useful and saved us a lot of time.

What's next for Cozy Koalas

A mask recognition system can be used in multiple other fields. For instance, airports, hospitals, quarantine centers, malls, schools and offices could reinforce their mask mandates without having to buy extra hardware. A software that can be used in conjunction with existing camera feeds would simplify this task and the analytics provided could also help those organizations better plan their resources.

In the technical aspect, a clear next step would be to incorporate face recognition into our model using FaceNet library (https://ieeexplore.ieee.org/document/9451684). This would essentially map the face images it gets from the cameras’ feed to identified individuals inside our deep convolutional network. Whether or not this feature will be used is up to the user’s discretion; however, it would be necessary for our system to automatically send a text message to the right person. Other next steps include increasing accuracy through a more balanced dataset (SMOTE balancing can only do so much...), adding distance measurements, and providing more analytics in the dashboard.