Heart rate detection via camera and machine learning while experiencing cloud scene.
Heart rate detection via camera and machine learning while experiencing mountain scene.
Testing webcam to read heart rate, and how it changes by listening to classical music.
"Creative computing" + contemplative visuals/audio to calm & de-stress the user by personalization via machine learning of biodata.
How machine learning can be used for artists to find novel ways to create with simple laptop or phone devices.
Slower heart rates are also beneficial to health as heart diseases are the leading cause of death globally.
Most people use laptops or smartphones that can become personalized to improve the heart condition.
We learned how visual and audio environments can become more creative and contemplative by adapting to the user's heart rate via machine learning.
By using machine learning and the digital camera of a laptop (or phone), the "creative computing" process could be integrated into any screen-/audio-based experience.
How to... We use the camera of the laptop and machine learning algorithms to detect the user's heart rate. Different visual and audio scenes are displayed to the user. The machine learning analyzes which scenes result in the lowest heart rate of the user. The machine learning then learns which combination of audio and visuals support the lowest heart rate. Then, the user experiences her/his/their "heaven" (aka visuals and audio calming the nervous system).
Built with... Unity, C#, digital camera of laptop
Challenges... To network the multiple devices and programs was a challenge. And to integrate the programming of the heart rate detection into Unity to personalize the screen and audio experiences.
Accomplishments that we're proud of... The approach is (one of) the first to personalize outcome data for calming and de-stressing visual and audio experiences by using heart rate biofeedback and machine learning.
What we learned... We learned how to efficiently divide the multitude of tasks, how to use the camera of the laptop as a heart rate sensor, and how to analyze the display of visual and audio environments based on biodata.
What's next for "NEHAN - Your Own Visual Heaven"... The further development of "NEHAN - Your Own Visual Heaven" will improve the learning algorithms to accurately detect heart rate and how it changes based on visuals and audio environments. Other biosensor could be used to detect for instance breathing patterns using the digital camera or a small thermal imaging camera for smartphones.
NEHAN project by Noah Picard and hannes bend