Folk Dance Classification with AI

Performance artists (folk dancers) performing the 'Khubaiti' folklore during SAAI's regional hub. MoCap was conducted indoors and outdoors.
The 'Khubaiti' traditional dances are often recognized as recreational, and celebratory, and they are used as regional identifiers.
Model trained on images captured during the regional hub. Further work involves motion capture via wearables and via motion sensors.
The 'Ardha' is a combination of singers, dancers carrying swords and a narrator. Performers dance shoulder to shoulder in two facing lines.

Inspiration

The work in this project aims to document folk dance in indigenous populations in the Arabian Peninsula. These forms of intangible heritage are inadequately understood and are not documented in labeled formats within existing multimedia archives. Since folk dances are strongly linked to local identity and culture, we were inspired to explore the different ways in which AI can identify, classify and perhaps generate visualizations of folklore dance in this project.

What it does

The machine is trained with videos of folk dance from different regions with the aim of classifying multimedia content based on clips of folk dance of varying resolution. Another component is to utilize the motion capture to recreate 3D models of motion in folk dances with samples of representative performers to be deployed in VR productions for traditional arts.

How we built it

The initial phase of this project is carried out using the pose analytics in Teachable Machines. Data was collected during the SAAI regional hub in Riyadh on Aug 21-22. Regional folklore performing artists worked with the participants to demonstrate traditional dances from different regions of the Arabian Peninsula. Teams were exploring the different ways motion capture can be considered and aligning the data acquired with the type of models that can be built for classification and generating digital rendering of such performancing via ML. In this phase, I used the videos and photos of the different traditonal dances to train the model via 50 epochs - a neural network term referring to the number of times each and every sample of the dataset is fed through the training model where one epoch is equal to one forward pass and one backward pass of all the training. Further work will involve motion capture to acquire high-resolution multimedia of the performing artists for several traditional dances. Datastreams from werables sensors as well as video capture will be integrated to create virtual experiences to view and interact in such performances.

Challenges we ran into

While we recognized that performing arts and in particular dance (e.g. folk dance) is an important element in intangible cultural heritage, we also realized that preserving, documenting, analyzing and visually understanding choreographic patterns is a challenging task due to technical challenges it involves in video capture and in motion-capture.

Accomplishments that we're proud of

During the SAAI Regional Hub in Riyadh, I had the opportunity to work with performing artists demonstrating folk dance from different regions in the Arabian Peninsula. I am proud of the collection of videos and motion clips that participants in the regional hub collectively shared (i.e. filming in indoor and outdoor contexts) which kickstarted this line of work - data engineering - for further research and development in AI-enabled folk dance heritage digitization.

What we learned

Learned more about AI models for pose analytics. Also, an important element that emerged is how choreography is a time-varying 3D process including dynamic co-interactions among different actors (i.e. in this case, folk dancers), emotional and style attributes, as well as supplementary elements such as the music tempo, the rhythm, traditional costumes (e.g. thoub).