-
-
pic1 model
-
pic2 weights and matrix
-
pic 3 results
About SenticWaveX
Overview
SenticWaveX is an advanced real-time AI-driven emotion recognition system optimized for Windows on Snapdragon. It leverages cutting-edge speech processing models, ONNX optimization, and real-time audio analysis to detect human emotions with high accuracy. Designed for low-latency execution, SenticWaveX applies confidence boosting, temporal smoothing, and signal quality enhancement to improve prediction reliability.
Key Features
- ✅ Real-time Audio Emotion Recognition
- ✅ Optimized for Windows on Snapdragon using ONNXRuntime
- ✅ Deep Learning-powered with WavLM (Wav2Vec2-based Model)
- ✅ Confidence Boosting & Emotion Smoothing Mechanisms
- ✅ Adaptive Noise Filtering & Audio Preprocessing
- ✅ Hardware-accelerated Execution using QNN Execution Provider
Technical Breakdown
1. Emotion Recognition Model
SenticWaveX employs WavLMForSequenceClassification, a Transformer-based model built on WavLM (a self-supervised learning model for speech representation learning). It utilizes:
- Wav2Vec2FeatureExtractor to convert raw waveforms into numerical features.
- Custom WavLM Model with Xavier weight initialization to improve training stability.
- Softmax-based classification to predict emotion categories:
- Angry 😠
- Happy 😊
- Sad 😢
- Neutral 😐
- Surprised 😲
- Fearful 😨
- Disgusted 🤢
- Angry 😠
2. ONNX Optimization for Snapdragon Acceleration
To ensure low-latency, efficient inference, SenticWaveX optimizes the deep learning model using ONNXRuntime:
- Model Conversion to ONNX Format:
- Uses
torch.onnx.export()to convert WavLM into ONNX format. - Defines dynamic axes for variable input sizes.
- Uses
- Execution Provider Selection:
- Uses QNNExecutionProvider if running on Snapdragon hardware.
- Falls back to CPUExecutionProvider when Snapdragon acceleration is unavailable.
- Uses QNNExecutionProvider if running on Snapdragon hardware.
- ONNX Graph Optimization:
GraphOptimizationLevel.ORT_ENABLE_ALLensures model execution efficiency.
3. Confidence Boosting & Emotion Smoothing
To improve prediction reliability, SenticWaveX incorporates post-processing techniques:
✅ Probability Transition Matrix
- Uses a 7×7 transition matrix to smooth emotion changes based on likelihood.
- Ensures emotions don’t switch erratically (e.g., angry → happy unlikely).
✅ Signal Strength & Acoustic Quality Factors
- Computes signal-to-noise ratio (SNR) to assess audio quality.
- Adjusts prediction confidence scores based on signal clarity.
✅ Temporal Consistency Enhancement
- Stores last 5 predictions to apply a moving average for stable output.
- Prevents jittery predictions when processing consecutive audio frames.
4. Real-time Audio Processing
SenticWaveX processes live audio using Sounddevice (sd.InputStream), ensuring seamless data capture with:
- Efficient Buffering Mechanism: Uses a FIFO queue for real-time processing.
- Adaptive Block Size: Captures 8,000 samples per block for smooth streaming.
- Multi-threaded Processing: Separate threads for audio acquisition & model inference.
5. Error Handling & Logging Mechanisms
The system is highly robust, with dedicated error management:
AudioDeviceErrorClass handles audio hardware failures.- Custom Logging System (
ColoredOutput) for:
- 📘 Status updates (
[STATUS]in blue) - ❌ Errors (
[ERROR]in red) - ✅ Success messages (
[SUCCESS]in green)
- 📘 Status updates (
6. Adaptive Noise Filtering & Preprocessing
To handle real-world audio distortions, SenticWaveX:
- Normalizes audio signals to prevent amplitude variations.
- Removes silent frames to focus on meaningful speech data.
- Estimates background noise levels to boost detection accuracy.
Conclusion
SenticWaveX is a high-performance emotion recognition system optimized for real-time AI applications on Windows on Snapdragon. It integrates deep learning, ONNX optimization, temporal smoothing, and real-time audio analysis to deliver fast, accurate, and stable emotion predictions.
🚀
Log in or sign up for Devpost to join the conversation.