Emotion Perception for Autism Support

• 

  An outline or roadmap for process development

Inspiration

Inspired by the need to support individuals with autism in understanding and processing emotional cues. Existing solutions lack real-time, multimodal emotion detection and personalized feedback. Aimed to create an accessible, technology-driven solution to improve social and emotional interactions.

What it does

Recognizes emotions from facial expressions, speech, and text inputs. Provides real-time feedback through visual or haptic cues using smartphones or wearables. Helps individuals with autism better navigate social scenarios by offering emotional insights and calming interventions.

How we built it

1)Models: Used TensorFlow/Keras for facial emotion recognition, Librosa/DeepSpeech for voice analysis, and Hugging Face for text sentiment analysis. 2)Hardware: Integrated cameras and microphones for input, and wearable devices for output feedback. 3)Backend: Flask/Django for processing, with WebSocket/MQTT for real-time communication. 4)Testing: Trained and validated models using datasets like FER-2013 for faces, RAVDESS for voice, and Sentiment140 for text.

Challenges we ran into

1)Data Integration: Combining multimodal data (facial, voice, text) seamlessly was challenging. 2)Real-Time Processing: Ensuring low latency in emotion recognition and feedback delivery. 3)Hardware Limitations: Constraints on processing power and cost-effective hardware for deployment. 4)User Interface: Designing a simple, accessible interface for diverse user needs.

Accomplishments that we're proud of

Successfully integrated facial emotion detection into a single system. Achieved accurate real-time feedback with minimal latency. Designed a user-friendly interface that provides actionable insights for users. Ensured privacy compliance and secure data handling throughout the system.

What we learned

The importance of multimodal integration for robust emotion detection and alos how technologies can be useful in various industries to integrate technologies.

What's next for Emotion Perception for Autism Support

Deploying the solution in classrooms, therapy centers, and homes.

Built With

• django
• librosa
• librosa-(audio-analysis).-flask/django-(backend)
• natural-language-processing
• nvidia-jetson-nano-(edge-processing).-wearables
• opencv-(ai-models).-hugging-face-(nlp-models)
• pytorch
• tensorflow
• usb-microphone.-raspberry-pi
• websocket/mqtt-(real-time-feedback).-hardware:-camera-(webcam-or-raspberry-pi-camera)
• websockets