Once there was a horse with unbelievable skill. The horse was called “Clever Hans” because it could read people’s minds. The horse could guess any number you were thinking of. For example, if you had the number 84 on your mind, Clever Hans would look at you, tap its foot 84 times, and then stop. How did Clever Hans do this? Of course, Clever Hans could not actually read your mind. But Clever Hans could analyze body postures. It saw the tension rising in your body when it started tapping, it noticed the slight jerk of your head when it reached your imagined number and stopped tapping. This happened a century ago. Today, machines can do this too.
What it does
It detects irregular head movements (< 2mm) as a sign of having correctly "guessed" a number you were thinking of.
How we built it
The proposed setup consists of a screen with a projection of an animated horse, a webcam to observe a person in front of this, and a Raspberry Pi to run the posture analysis.
Currently, we have coded a two-stage implementation:
- Detect head posture movements like pitch, roll, and yaw in a video using a pretrained model like RealHePoNet or img2pose and save them as arrays
- (Optional) Visualize those movements over time
- Use an anomaly detection library to train an unsupervised ensemble of classifiers that detects irregular head movements
- Mark the detected outliers as potential head jerks
Challenges we ran into
- Identifying irregular head movements (< 2mm) in a constantly moving/breathing person.
- Multiple outliers are present while a person is being recorded. It is difficult to know which is the exact outlier that we need. This is because the one with the highest "outlier score" (the global maximum) may not be the true head jerk.
- There is a lack of labeled data present to use a supervised anomaly detection algorithm instead
Accomplishments that we're proud of
- Linking together state of the art head posture recognition and anomaly detection algorithms
- Creating the foundation for a "mind-reading" application
What we learned
- Using pretrained models for head posture recognition
- Using anomaly detection to detect outliers
- Data visualization
What's next for Smart Hans
- Use Electromyography (EMG) to detect how motor nerves can be used to detect the head jerk
- Create rich, labeled data to improve the anomaly detection algorithm with supervised learning
- Incorporate more features like the movement of other facial features or body parts
- Explore further application possibilities beyond guessing numbers.