Bahen's Crossy Road

dean yip
Streamlit

Inspiration

Whether we’re late to class or meeting up with a friend, dangerous jaywalking the crosswalk between the Bahen Centre and Galbraith building is all too common for us UofT engineering students. With inspiration from the show “Squid Game,” we made Bahen’s Crossy Road, an interactive game that catches jaywalkers red-handed (and hypothetically automatically deducts meal plan money.

What it does

Bahen’s Crossy Road is a gamified version of the infamous crosswalk between the Bahen Centre of Information Technology and the Galbraith Building. Inspired by the show “Squid Game” on Netflix, Bahen’s Crossy Road lets players compete against each other when crossing the road. The goal for every player is to cross the road first without being eliminated. Players are eliminated whenever they are moving during a red light and are free to move during a green light. When the traffic light, in this case, a figurine of Dean Yip says green light, players are free to move until he turns around and says red light. Scores and current neural network processes are streamed in an interactive web app. When someone crosses the road, the game ends, and their score is increased on the leaderboard.

How we built it

We built Bahen’s Crossy Road using an ESP32 with a camera, Python and streamlit which displays the leaderboard and handles all the game, detection, and streaming logic. We also used redis for storing to increment player wins and handle race conditions with fetching data from the esp web server. We leveraged cloudflare-based gemini to create dynamic responses to different player codes winning.

Challenges we ran into

One of the biggest challenges we faced was an issue with our 3D printing request. When we went to collect our 3D-printed part, we were informed that our submission had been rejected some time ago, but we had never received any notification about it. This unexpected setback forced us to rethink our design at the last minute. To adapt, we quickly came up with a solution that we used a paper box to replace our 3D printing to keep the project on track.

We had difficulty implementing a neural network that could detect multiple skeletons, and an algorithm that can efficiently and accurately discard noise and calculate moving skeletons.

Accomplishments that we're proud of

Our team is really proud of how we integrated the ESP32 webserver technology with our Python code in order to send real-time camera data onto a website, and then into the Python code. This allows us to not rely on a microcontroller to process the data, and rather, leave the task to a much better computer to analyze the image data to see if any players have moved or not from their original position.

We are also proud of how we managed to determine each player’s motion status through utilizing AI, and a filtering system that compares initial points to new points to capture movement.

What we learned

We learned: How to work with DNNs and tensorflow Streamlit and cloudflare Using the ESP32-CAM and sending packets via wifi to a python app What's next for Bahen's Crossy Road Looking ahead, we see the potential to develop the Bahen’s Crossy Road beyond just a video game and transform it into a real-life system that can help prevent jaywalking. By integrating our motion detection technology with traffic cameras, we could create a system that monitors pedestrian movement at crosswalks and provides real-time alerts when someone crosses unsafely. Furthermore, this could be enhanced by AI-powered traffic monitoring, which can analyze pedestrian behaviors, detect patterns, and potentially work with city infrastructure to improve the road safety.