(wrnch) Gesture2Command

Body language recognition and interpretation

Comment 2

Inspiration

It's cool to operate a computer without typing on keyboard or moving mouse.
What if a few body gestures could help you do most of the work?!

What it does

Our python scripts record and recognize user's body language and send corresponding command to computer.

How we built it

  • Use wrnchAI models to return joints coordinates of human body
  • Label the frames(joints cordinates) and create dataset
  • Train the neural network and save the model
  • Use camera the record the user's gesture and recognize it

Challenges we ran into

  • We had a big problem with the Python version compatibility, making wrnchAI APIs compatible with Python 3. So we have to set up two environment, store temp data and making two scripts run in parallel.
  • Create our own dataset can personalize the design but not large enough. Our model could be not accurate.
  • However so far it appear good.
  • We still have a lot of awesome ideas that are not implemented yet. Given more time we could do it much better.

Accomplishments that we're proud of

  • Shot all the troubles!
  • Finally work out a demo!

What we learned

  • Using wrnchAI APIs
  • How to design a project when there is not enough time.

What's next for Gesture2Command

  • Instead of taking seperate frames as input, we will use the whole video as input data.
  • Create larger dataset of videos.
  • Add LSTM to train the model.
  • Design generalized APIs to the client(including a beautiful GUI but not the shell).
  • Currently we only recognize the classification without adding specified actions after this.

Built With

Share this project:

Updates