Skelichen

Inspiration

In the 1970s, chamber orchestras were revolutionized by the idea of the blind audition. Prior to that point, an audition had to be performed in front of a judge, which introduced the performer's identity, inadvertently or otherwise, into the list of criteria on which the performance was judged. Things like the applicant's race and gender influenced the judge's evaluation of the performance, almost exclusively to the advantage of Men and White people. From the 1970s to the present day, however, blind auditions have largely removed the performer's identity from the equation, and this has substantially increased the number of Women and racial minorities in chamber orchestras.

Unfortunately, this was not possible for visual arts -- for figure skating, or dancing, or gymnastics -- this was not possible at the time. Evaluating performance in these fields necessarily involves seeing the performers' movements, which, until recently was not possible without seeing the performer. In many of these fields, performers specifically still have to deal with implicit and explicit biases affecting perceptions of their performance. The top levels of figure skating and gymnastics are overwhelmingly White and East Asian, owing in large part to the fact that other racial groups receive systemically lower scores, especially at lower levels of competition.

What it does

Skelichen detects the position of a human body throughout a video and tracks it throughout the video. This allows us to evaluate the body's movements and provide that to a judge -- either for a competition or another talent evaluation -- with the identity of the performing individual removed. This would allow for a race- and gender-blind evaluation process, and would be a first step toward bridging the access gaps in competitions that depend on a judge's subjective evaluation.

How we built it

We implemented OpenPose's pose-detection technology to detect the pose of figure skaters in videos. Figure skating was chosen as a good proof-of-concept case due to the fact that figure skaters largely stay upright, which is key to OpenPose's efficacy.

We also used video of figure skating to train a model that detects and classifies the actual move that is being performed in a video.

Challenges we ran into

On a consumer-grade laptop computer, OpenPose's pose detection has a very low framerate, and thus inaccurately represented the skater's body positions throughout the video. As such, the video seen here had to be slowed down to match the framerate that our devices could handle, and then sped back up.

Accomplishments that we're proud of

Managed to classify, detect, and count the number of turns performed by a figure skater despite turns being performed in non-standard positions.

What's next for Skelichen

Extending this method to other sports; if we could have appropriately tracked the body positions associated with the flips performed in Gymnastics, for example, this technology could be similarly powerful in that field. This would have required us to either implement or develop a pose detection algorithm that correctly identifies humans when they are in non-standard poses.

With more computing resources -- particularly a strong GPU, which none of our laptops had -- this analysis could be conducted in a much shorter time, approaching real-time.

If time had permitted, we'd have liked to analyze more data to better train our classification models.