CUTY- your AI assistant robot for COVID-19

CUTY is an AI-based assistant robot. She helps you to check your health in real-time. by looking into your face and hearing your voice !!!

Comment 1

Inspiration

During this pandemic, we always wish, if older/blind community have a companion to check their health status. They are more vulnerable to this disease and are little known to modern mobile applications/web applications etc. To give them an alert when the heartbeat is above 100 bpm if they cough frequently give a notification and check their emotional status(so that they won't feel depressed). We made this bot for anyone who want to check their health-status and also get their questions answered, they can simply ask to CUTY. We made This AI partner cheap and affordable to the global community.

What it does

The bot constantly analyzes your face. Using computer vision and deep learning it will identify the strongest emotion and the current heart-rate,we set the heart-rate range 60-100 if it exceeds normal, cuty will identify it, along with that using an analog inbuilt microphone it recognizes the cough and counts it. Everything is save it as a plain text file and with a push of a button, CUTY will say, Good morning sir, from the facial analysis you look happy and have a score of 8. Your heart rate is 62 beats per minute and it is normal, you have a moderate cough, and the cough count up till now is 15. You can connect CUTY with your bluetooth audio system. CUTY is completely offline, everything happens in the edge, which means that it is more secure. She can be mounted anywhere, on your desk, the table you are working, or near your laptop/TV.

How we are building it

Hardware- AI single Board computer, 5 MP camera, analog/bluetooth audio system, 10000 mAh battery, 3.3V to 5V DC-DC boost converter, push button

Programming language- Python

ML architectures/Frameworks -TensorFlow, Caffe

API - Keras

Libraries -TensorFlow, dlib, opencv, scipy(signal processiong) Deep learning techniques- Face detection(caffe), emotion recognition(Keras with TensorFlow backned), Sound detection(Keras with TensorFlow backend)

Program specifications

Heart-rate detection

Principle: detecting changes in sking color due to blood circulation and calculate heart-rate

  1. Face detection using dlib library and get the Region Of Interest(ROI)

  2. Apply a band pass filter to eliminate a range of frequencies

  3. Average colour value of the ROI calculated and pushed to a data buffer

  4. Apply Fast Fourier Transform to the data buffer. Highest peak is the heart-rate

Cough detection

Principle: convolutional neural network(CNN) with Keras using TensorFlow backend

  1. Collected sound data: https://voice.mozilla.org/en/datasets, https://urbansounddataset.weebly.com/urbansound8k.html, https://github.com/hernanmd/COVID-19-train-audio/tree/master/not-covid19-coughs

  2. Used transfer learning on the VGG-16 architecture Pre-trained on YouTube-8M for audio recognition

  3. Save the keras model and used for real-time prediction

Emotion recognition

Principle: CNN with Keras using TensorFlow backend

  1. Dataset, FER2013 from Kaggle

  2. Construct CNN with Keras using TensorFlow backend

  3. Train the model from the given dataset

  4. Face detection using Caffe based pre-trained deep learning model

  5. Real-time emotion recognition and plot animated matplotlib graph from the output.

Challenges we ran into

  1. Building the hardware system is one hell of a job. The entire program should run with a push button, means that after GPIO configuration three program ( cough recognition and counting, emotional recognition, heartbeat detection) should be multiprocessing and save the results in a single text file so that AI text to speech engine can convert to audio.

  2. To make a wonderful case for the system also a challenge we faced.

  3. Demo video editing is a tough job since we used open-sourced video editing software such as Kdenlive and olive for editing.

What have you built during this weeekend

Almost everything we made happens during this weekend. From programming to prototype, we connected CUTY with Bluetooth headset. Now you can hear CUTY even you are busy in your schedules, we made a customized AI- Single board computer that helps her to work fast. Built a wonderful case so that it looks neat. Updates: We made her small enough to fit in the palm, integrated Bio-BERT NLP engine, that is now she can hear you and reply to your query.

The value of the solution after the crisis

She can help anyone who needs to check their physical health, emotional health in real-time. This helps keep track of their overall health status during the regular work. We made CUTY exclusively for the community.

How can we scale it

Since SBC's are cheap and has a plenty in the market, same in the case of batteries, audio system and the case. Funding will helps us to make thousands.

What's next for CUTY- your AI assistant

  1. Make a wonderful body for CUTY, we made the basic prototype on aluminium.

  2. Integrating Alexa/Siri like systems in to CUTY

  3. Integrating Natural Language Processing(NLP) & Natural Language Understanding(NLU) in CUTY(updates: Integrated Bio-BERT in to CUTY)

  4. Since CUTY can be connected to internet via WiFi and Ethernet, she can also be updated via online. We need to use a secured service for that. For that reason we didn't made her online yet, everything from deep learning prediction, analysis, text to speech happens in offline and in the edge but in future we will connect her to a secure server

  5. CUTY uses Reinforcement Learning(RL) to improve herself, but this might be a little tough to handle, considering her computational capacity. Optimized Q-learning techniques are also under research.

  6. Using 3D Convolutional Neural Network for detection of heart rate. As of now CUTY uses computer vision and signal processing techniques such as Fast Fourier Transform(FFT) to detect heart rate. She is accurate as we tested against standard results, but the accuracy can be improved by integrating 3D CNN along with computer vision.

Reference

1.Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. In The 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 213-224

  1. Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. 2016. Youtube-8M: A large-scale video classification benchmark
  2. J. Amoh and K. Odame, "Deep Neural Networks for Identifying Cough Sounds," in IEEE Transactions on Biomedical Circuits and Systems, vol. 10, no. 5, pp. 1003-1011, Oct. 2016
  3. I. D. S. Miranda, A. H. Diacon and T. R. Niesler, "A Comparative Study of Features for Acoustic Cough Detection Using Deep Architectures*," 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 2019, pp. 2601-2605.
  4. J. Amoh and K. Odame, "DeepCough: A deep convolutional neural network in a wearable cough detection system," 2015 IEEE Biomedical Circuits and Systems Conference (BioCAS), Atlanta, GA, 2015, pp. 1-4.
  5. S. Shin, T. Hashimoto and S. Hatano, "Automatic Detection System for Cough Sounds as a Symptom of Abnormal Health Condition," in IEEE Transactions on Information Technology in Biomedicine, vol. 13, no. 4, pp. 486-493, July 2009.
  6. Facial Expression Recognition Using Convolutional Neural Networks
  7. S. Li and W. Deng, "Deep Facial Expression Recognition: A Survey," in IEEE Transactions on Affective Computing.
  8. M. Soleymani, S. Asghari-Esfeden, Y. Fu and M. Pantic, "Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection," in IEEE Transactions on Affective Computing, vol. 7, no. 1, pp. 17-28, 1 Jan.-March 2016.
  9. Rahman, Hamidur & Ahmed, Mobyen & Begum, Shahina & Funk, Peter. (2016). Real Time Heart Rate Monitoring from Facial RGB Color Video Using Webcam.
  10. Poh, Ming-Zher, Daniel J. McDuff, and Rosalind W. Picard. “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation.” Optics Express 18 (2010): 10762. ©2011 Optical Society of America.

Built With

Share this project:

Updates