Two of the most ubiquitous aspects of life in lockdown are video calls and online games, and we were inspired to create something that would bring the two together. CopyCat is based off of a teambuilding exercise, where players successively copy one another's actions, adding a new action each time. We thought it would be an interesting and rewarding challenge to use computer vision to recognize and detect similarities in these actions.
What it does
Players join the game by opening a web app and joining a room, which is similar to a video call. At the start of each round, the current player must correctly perform all previous actions in the correct sequence. The video is recorded and sent to the PoseNet OpenCV model, which calculates the similarity between the video and the previous sequence of actions. If the player successfully performs all previous actions, they get to add their own action to the sequence. Otherwise, the player is eliminated. In either case, the game moves on to the next player. The game continues until there is only 1 player remaining.
How I built it
Our back-end consists of a Game class, which stores a video of the previous poses. It processes the video by downloading it from the link, unzipping it if necessary, and extracting the newest action before passing it to the computer vision model.
The computer vision pose detection algorithm is built using Tensorflow and OpenCV in Python, alongside the Posenet dataset to establish the key points of the body. Calculation of deviation uses dynamic time warping library DTAIDistance to ensure that actions are compared accurately regardless of speed or time differences between videos.
Challenges I ran into
Our biggest challenge was connecting the various components of our application. While our front-end, back-end, and computer vision model work well individually, we had trouble sending messages between them. As such, the user experience is still unpolished. We also faced many issues with running our code, as the computer vision model in particular had many dependencies that were sometimes incompatible with those already installed on our computers. Some of us had to go as far as updating and restarting their laptop before they could run the code.
Accomplishments that I'm proud of
We were very proud of the algorithm we used to effectively calculate the similarity between actions. We experimented with several alternatives, such as cosine similarity and cross-correlation, but ultimately went with an algorithm that combined multiple approaches and normalized the images. This reduced the influence of body size, speed of movement, and location in the frame on the similarity of actions and improved accuracy.
We were also proud of how we built a project that was both interesting to us and useful for other students during lockdown. University is tough and staying at home can get lonely, so forging connections between people through a game form their childhood is rewarding for us as well.
What I learned
What's next for CopyCat
Currently, we only support one game room. When using the website, all users join the same video call. In the future, we would like to let users create multiple rooms, so that more people can play our game at the same time.