Developer Frustration Detection System

Detects developer frustration in real-time using OpenFace facial recognition and keystroke analysis with 75% accuracy, triggering AWS Nova conversational AI for immediate support.

Comment

Inspiration

Software developers frequently experience frustration during debugging, compilation errors, and complex problem solving. Prolonged frustration can reduce productivity, increase cognitive load, and negatively affect mental well-being.

Most development tools focus only on code correctness and performance but do not consider the emotional state of the developer.

We wanted to build an emotion-aware development support system that can detect developer frustration in real-time, analyze behavioral and emotional signals, provide supportive debugging assistance, and encourage healthier coding practices.

Our vision is to build emotionally intelligent development environments that support both developer productivity and well-being.

What It Does

The Amazon Nova Powered Frustration Detection System is an AI-powered system that detects developer frustration during coding sessions using:

  1. Keystroke Dynamics Analysis - Captures typing speed, backspace frequency, pause intervals, and burst typing patterns

  2. Facial Emotion Recognition - Uses computer vision with OpenCV, MediaPipe, and TensorFlow to estimate emotional state

  3. Multimodal Fusion Engine - Combines both signals using LightGBM to estimate frustration probability

  4. CodeBuddy AI Assistant - When frustration is detected, launches an interactive assistant powered by Amazon Nova

  5. Real-time Support - Provides debugging suggestions and encouraging feedback through a lightweight popup interface

The system detects frustration with 75% accuracy and provides immediate AI-powered support to help developers recover quickly from debugging difficulties.

How We Built It

System Architecture

The system follows a multimodal detection pipeline:

Keystroke Logger + Facial Emotion Detection → Feature Extraction → LightGBM Fusion Model → Frustration Probability → CodeBuddy Popup Assistant → Amazon Nova AI Response

Technology Stack

Programming:

  • Python

Machine Learning:

  • LightGBM
  • Scikit-learn
  • Joblib

Computer Vision:

  • OpenCV
  • MediaPipe
  • TensorFlow

AI Integration:

  • AWS Bedrock
  • Amazon Nova

Interface:

  • Tkinter popup assistant

Key Components

1. Keystroke Logger Captures behavioral typing signals including typing speed, backspace frequency, pause intervals, and burst typing patterns.

2. Facial Emotion Detection Uses computer vision to estimate emotional state based on facial expressions.

3. Multimodal Fusion Engine A LightGBM model combines keystroke behavioral features and facial emotion probabilities to estimate frustration probability.

How We Built It

System Architecture

The system follows a multimodal detection pipeline:

Keystroke Logger + Facial Emotion Detection → Feature Extraction → LightGBM Fusion Model → Frustration Probability → CodeBuddy Popup Assistant → Amazon Nova AI Response

Technology Stack

Programming:

  • Python

Machine Learning:

  • LightGBM
  • Scikit-learn
  • Joblib

Computer Vision:

  • OpenCV
  • MediaPipe
  • TensorFlow

AI Integration:

  • AWS Bedrock
  • Amazon Nova

Interface:

  • Tkinter popup assistant

Key Components

1. Keystroke Logger Captures behavioral typing signals including typing speed, backspace frequency, pause intervals, and burst typing patterns.

2. Facial Emotion Detection Uses computer vision to estimate emotional state based on facial expressions.

3. Multimodal Fusion Engine A LightGBM model combines keystroke behavioral features and facial emotion probabilities to estimate frustration probability.

Example output: P_keystroke: 0.93 P_face: 0.12 P_total: 0.67 frustrated: True

4. CodeBuddy AI Assistant When frustration is detected, the system launches an interactive assistant called CodeBuddy with chat-based debugging assistance and encouraging developer feedback.

5. Amazon Nova Integration Powered by Amazon Nova through AWS Bedrock, providing natural language debugging guidance and troubleshooting advice.

Challenges We Faced

1. Real-time Processing Processing facial expressions and keystroke data in real-time required optimizing the data pipeline to handle multiple inputs simultaneously without latency issues.

2. Model Accuracy Achieving 75% accuracy required extensive testing and fine-tuning of both the OpenFace/MediaPipe model and the keystroke dynamics model, ensuring they complemented each other effectively.

3. Integration Complexity Combining multiple technologies (OpenCV, MediaPipe, LightGBM, AWS Nova) into a seamless system required careful architecture planning and rigorous testing.

4. Data Privacy Handling sensitive facial data required implementing proper security measures. We ensured facial data is processed locally and no biometric data is permanently stored.

What We Learned

  1. Emotional AI is complex - detecting human emotions requires multiple data points for accuracy
  2. Real-time processing is challenging - optimizing for performance without sacrificing accuracy is a delicate balance
  3. Developer wellbeing matters - the problem of developer burnout is real and technology can help address it
  4. Integration is key - making multiple technologies work together seamlessly is often harder than building individual components

Built With

Share this project:

Updates