Zoom Boxing

Inspiration

Really Wanted to Punch my Friend

What it does

Allowed me to Box my friend without legal repercussions

Brief Explanation on how game works

• One person can create a "Zoom" Meeting and someone can join a particular zoom meeting through the room id
• Then the 2 players can fight it out by throwing punches at each other (with good form of course, you should read the How to Play tab before starting)
• Players can dodge the attack either to the left or to the right
• First person to lose all their health is the worse boxer

How we built it

WebRTC + Pose Estimation

Detailed Explanation + Challenges & "Solutions"

WebRTC

Pose Estimation - Punching + Ready Position

• Most punch detectors are able to do this only at an angle (essentially anyting but punching towards the camera) since they recognise punches by calculating arm angles (angle between wrist point, elbow point and shoulder point)

• However for our use case using arm angles is not a valid solution since we are punching towards our camera / webcam (make us big sad)

• We solved this by marking detecting keypoints in pose estimation instead and determine if the boxer is in a ready position by checking various conditions:

1. Distance from Hand from Nose (High Guard Ready Position)
2. Distance from Hand to Shoulder (Basic Guard Ready Position)
3. Presence of all Arm Key Points, i.e Wrist, Elbow and Shoulder ... And More

• We then log punches by checking for frames where the boxer's wrist and elbow disappears for a split second after being in a ready position.

• Note that we decided to tackle this like a Finite State Machine, alternating between ready and not ready position because we want to prevent a particular user from throwing multiple punches in quick succession (That would defeat the purpose of the game right :D)

Pose Detection - Hit Detection

• We wanted players to be able to dodge attacks depending on if it was launched to the left or to the right

• We essentially dealt with this by going to the exact frame when your opponent "punched you" and take the pose estimation keypoints at that exact moment

• We then make the assumption that your nose is the centre of your head and track your node keypoint's relative position on the screen to determine if you got "hit" or not

What's next for Zoom Boxing

More Dynamic Boxing Experience, maybe make them feel real pain? o.O

Built With

• javascript
• react
• tensorflow