Pose-estimation algorithms have improved exponentially in runtime and accuracy. Because of this, classifying movements quickly is much more feasible now that it has ever been. In other words, given a video of any person, we could classify his or her actions without the need for external hardware like a Kinect or Wii Sensor. There are several applications to this. For instance, one may be able to measure the harmful movement patterns of a human being. Imagine being able to control devices with your movements.

What it does

The application records a dance performed by

How I built it

The building of our application spanned across phases beginning with discussion of ideas amongst like-minded people followed by Brainstorming. The application contains a front-end developed by Android Studio that records the learner's dance video and stores the recording in Google Firebase followed by an application layer built on REST API using Flask that performs pose estimation on videos and scores cosine similarity of dance movements based on Time Dynamic Warping.

def frame_cost(a,b):
    a_fr = frame_vec(a)
    b_fr = frame_vec(b)
    total_cost = 0.0
    total_conf = 0.0
    for st in lines:
        for end in lines[st]:
            a_vec = np.array([ a_fr[st]['x']-a_fr[end]['x'], a_fr[st]['y']-a_fr[end]['y'] ])
            b_vec = np.array([ b_fr[st]['x']-b_fr[end]['x'], b_fr[st]['y']-b_fr[end]['y'] ])
            if np.any(a_vec) and np.any(b_vec):
                conf = 1 - math.fabs((a_fr[st]['conf']-b_fr[st]['conf'])*(a_fr[end]['conf']-b_fr[end]['conf']))
                cost = cosine(a_vec,b_vec)*conf
                total_conf += conf
                total_cost += cost

    return total_cost/total_conf

Challenges I ran into

Starting out was indeed a challenge, but we overcame it by challenging each other's ideas. Major challenge was in comparing videos, which are impacted by their temporal characteristics and we overcame it by using Dynamic Time Warping concept for continuous temporal signals and integration of data stored in Firebase with application instance in Google Compute Engine, which was overcome using REST API using JS and Flask.

Accomplishments that I'm proud of

  1. Extensively using Google Cloud Platform and Google API and integrating Firebase raw storage with application instance in Google Compute Engine.
  2. Using pose tagging to tag the postures of the body a person learning dance and comparing similarities between the learner's video and compared video using cosine similarities of posture positions considered as vector of feature points.
  3. Using Dynamic Time Warping to compare two video signals along with their temporal features in real-time.

What I learned

New Technologies like pose tagging, dynamic time warping, practical application of cosine similarity, Firebase, Google Compute Engine, and Flask, few of which I may not have explored otherwise. Topped with Self-Motivation, Team Building, Creativity, Quality, Security and meeting deadlines.

What's next for BustAMove

  1. We are planning to utilize the concept and logic of the application to expand it to help people be fitter, stick to healthy lifestyle, achieve greater by rewarding them points which can act as a method of inspiration as they might be able to redeem it at shopping outlets and hotel. Special acknowledge to pocket points for helping us with this innovative concept.
  2. Help industries in promoting their products and enhancing their business.
  3. Facilitate professors in enhancing the outcome of their class by rewarding attentive students.
  4. An idea of converting this application to an e-shopping portal like Amazon, ebay or the likes of paytm in India where people can buy commodities using the points earned is a main target behind our application.
Share this project: