Robotics-Based Action Observation Therapy (AOT)

Robot teaches learning user how to perform action through incremental changes from their course of action.
Guardian teaches the robot the action for the session.

Background

Children with cerebral palsy often have worse gross motor skills than their non-affected counterparts. Motor skills deficiencies lead to difficulties in coordination and everyday tasks. Physical therapy is a means of improving motor skills [1], [2] – a technique called action observation therapy (AOT) specifically has shown to cause improvements. AOT is the act of observing and imitating actions to activate the brain’s mirror neurons [3] .

So what's the gap?

Physical therapy is:

Expensive over a long period of time
May require child to be in a new environment with new people, causing stress
Not available on-demand for many

How BracketBot Helps

We propose using the BracketBot to aid children with Cerebral Palsy with actions that involve gross motor skills, without the need for visiting specialists. Any parent or household member can perform an exemplary action that they want the child to learn, after which they are not needed for the session. The child then attempts the motion. The BracketBot then analyzes the child's movement and provides incremental corrective feedback by demonstrating a sequence of movements that gradually transition from the child's current motion toward the exemplary action

We initially focused on ball-throwing due to its proven efficacy in improving coordination in children with Cerebral Palsy [4], before we adapted the project to allow for any arm-based motion.

Why does this solve the gap?

The BracketBot's incremental movement changes allows the child to undergo AOT in a straightforward manner that does not overwhelm them, while avoiding the aforementioned issues with physical therapy. This means families without easy access to physical therapy, or parents who are busy at work most of the day, don't have to sacrifice their children's motor skills development.

How we built it

MediaPipe and OpenCV to track arm movements - translated to robot's position using Numpy
MoveIt IK solver translates positions to joint angles
Robot's motion is visualized in Rviz
A scoring system compares camera input with recorded correct motion and adjusts robot's motion

Challenges we ran into

Integrating different pieces of software (ROS2, Gazebo, Rviz) led to many challenges, especially for making sure dependencies' version were consistent.

Accomplishments that we're proud of

Despite not having access to a physical BracketBot, we implemented our project fully through simulation and mimicked the camera feed of the BracketBot as a proof of concept.

What's next?

Deploying and testing our project on a real BracketBot
Improving its adaptability across a wider range of motions
Seeing what actions cause edge cases for troubleshooting
Leveraging BracketBot features such as microphone and extra camera feeds for improving performance and interactions

[1] Treatment Effects of Upper Limb Action Observation Therapy and Mirror Therapy on Rehabilitation Outcomes after Subacute Stroke: A Pilot Study [2] Action observation therapy in pediatric cerebral palsy: an fNIRS-based investigation of cortical responses and balance—a randomized controlled trial [3] https://www.physio-pedia.com/Action_Observation_Therapy [4] Effect of Ball Throw and Catch Game on balance and co-ordination in children with Cerebral Palsy