Overcrowding and waiting in lines is always a pain - it might even be dangerous in light of the current pandemic. If people could know the occupancy of buildings in real-time, they would be able to make smarter decisions about where to go, avoiding crowds, lines, and frustration. In addition, with past data about the occupancy of buildings, people could also identify trends to help them plan ahead for ideal times and/or locations to visit.
What it does
Given a video feed (usually from a security camera), Occupansee calculates the number of people inside a building and then displays the current occupancy and other related statistics on a webpage in real-time.
How I built it
Backend To detect people in the video feed, I used the YOLO Convolutional Neural Network algorithm trained on CoCo datasets, and OpenCV for image processing and other computer vision related tasks. I devised a method to calculate occupancy for larger buildings by taking the difference between the number of people entering and exiting. Then, I created an algorithm to determine which side of the frame a person exits from by tracking their previous locations and calculating which side they are closest to using their coordinates.
Database I stored the data in a Firebase Realtime Database. I chose to use the date and time of when the data was created as keys so that I could efficiently query the data by time later.
Server I created a server with python that queried data from the database and sent it to the front end via websockets. Additionally, I created functions to calculate hourly, daily, and weekly statistics as well as search and retrieve data by time frame.
Front End I built the front end using React to handle components and synchronization, and Chart.js for plotting the data on graphs. There is a little bit of CSS sprinkled in as well.
Challenges I ran into
• I had to configure python virtual environments, download computer vision and machine learning packages, and install libraries for the database, server, websockets, and React. Ensuring that everything was coordinated and working properly was difficult, confusing, and frustrating at times.
• My original prototypes for the people counting algorithm were unbelievably slow, nowhere near real-time. I optimized it in many ways, including reducing the number of times people were being detected, skipping frames, and experimenting with different object detection methods.
• Conditionally and synchronously rendering different instances of the same class component in React gave me a lot of trouble, but I solved it by using a special keyword for the passed in props and then putting the time-based function in a larger container component.
• I had to figure out how to pack the data (originating from python) into a format so that the React frontend could unpack and display it.
• I initially tried to make the server in Node, but I was running into a lot of issues with both the front end and the database, so I reverted back to python.
• In the beginning, I wasn’t able to grasp what the overall structure of the project should look like. My mentor helped me a lot with understanding it.
Accomplishments that I'm proud of
I am extremely proud of my resilience through the many pitfalls and obstacles I faced when creating Occupansee. I'm also proud that I genuinely became a better programmer, engineer, and innovator through this project. But more than anything else, I am proud of the fact that I was able to turn my vision into something tangible.
What I learned
• How neural networks (including convolutional) work, especially in the context of computer vision.
• How backends, frontends, and databases communicate with one another, and the different types of frameworks for doing so.
• Different ways to design and implement the flow of information, both in algorithms and between parts of an app.
• The overall structure of an application from top to bottom, and the functions and interactions each part.
• How to configure environments and utilize different types of computer tools for specific tasks.
What's next for Occupansee
• Obtain access to more security cameras, perhaps with cooperation with companies.
• Develop geographic and mapping features to provide further insight into traffic and congestion.
• Implement different object types (animals, vehicles, etc.) to enlarge the scope of Occupansee’s possible uses.
• Optimize the performance of the detection algorithm with CUDA (requires a NVIDIA Gpu).
• Expand to outdoor areas such as parks, sports fields, roads, etc.