Can computers be creative? The recent revival of recurrent neural networks (RNNs) influenced a wave of generative techniques that has often be labeled as creative. From creating excerpts of text written in the style of Shakespeare, to making a picture look like a Van Gogh, neural networks applications have been the basis of a breath of work in computational creativity. Dance is one sector of creativity that has seen limited research. If a computer could generate different dance routines, it could give inspiration to dancers, create entertainment, and perhaps semi-automate the creation of dance games such as Just Dance.
Think about the last time you watched a dancer; it's amazing how they match their movements to the beat and form dynamic shapes. I have been learning about neural nets and their generative capabilities, and wanted to combine this power with my hobby of breakdancing.
What it does
Right now I collect dance data in the form of (x,y, z) coordinates for joint positions to a csv using a Kinect depth sensor. I then use this data to train an LSTM RNN network using a mixture density model. The model has not had much success yet for forming human shapes -- I think my model needs work!
How I built it
I used pyKinect, a wrapper library for python to interface with the Kinect to collect data. I worked on the model in tensorflow based on Graves Handwriting model.
Challenges I ran into
I had some difficulty with matrix manipulations in tensorflow for the mixture density network implementation.
What I learned
I have learned some basics of tensorflow, and some more best practices for training and creating neural nets!
What's next for Dance RNN
I will keep working on this to get it up and running.