AI-Powered Job Interview Coach

The AI-Powered Job Interview Coach is an innovative web application designed to help users practice for job interviews using AI. The platform simulates an interactive interview environment where users can respond to role-specific questions, receive real-time feedback on their performance, and track their improvement.

Inspiration

We wanted to build a tool that helps job seekers prepare for interviews in a way that's accessible, interactive, and based on real-world data. By utilizing AI, speech analysis, and facial recognition, we aim to provide personalized feedback, simulate real interview scenarios, and improve user confidence in interviews. This project is inspired by the need to improve interview readiness through technology.

What it does

• AI Interview Simulation: The platform asks users questions tailored to the job role they select and evaluates their responses.
• Speech Analysis: The system analyzes user speech for clarity, tone, confidence, and filler words, offering constructive feedback.
• Facial Expression Tracking: Using camera input, the system tracks users’ facial expressions and body language, providing feedback on engagement and professionalism.
• Real-Time Feedback: After each response, users receive feedback on their speaking style, body language, and answer quality to improve their interview skills.
• Customizable Mock Interviews: Users can choose their desired job role and difficulty level to simulate the most relevant interview experience.
• Progress Tracking: Users can track their progress over time, reviewing previous responses and feedback.

How we built it

We used a combination of frontend and backend technologies, along with several APIs and AI models to build the app:

• Frontend:

  • React.js for building an interactive user interface.
  • Web APIs for capturing voice and webcam input for speech and facial expression analysis.

• Backend:

  • Flask as the web framework to handle API requests.
  • Google Cloud Speech-to-Text API for converting speech into text.
  • Google Cloud Text-to-Speech API to convert interview questions into voice.
  • MediaPipe for facial expression analysis.
  • Firebase for storing user data and feedback.

• Deployment:

  • The app is deployed on Google Cloud Run (backend) and Vercel (frontend), ensuring scalability and high performance.

Challenges we ran into

• Speech Recognition Accuracy: We faced challenges with speech-to-text accuracy, especially in noisy environments. Tuning the speech recognition model to handle various accents and speech patterns took more time than expected.
• Real-Time Feedback Integration: Providing real-time feedback based on voice, facial expressions, and text input required integrating multiple AI models simultaneously, which sometimes led to delays or performance issues.
• Facial Expression Analysis: Capturing and analyzing facial expressions in real-time with consistent accuracy, especially in varying lighting conditions, posed challenges in fine-tuning the model.

Accomplishments that we're proud of

• Successfully integrating multiple AI components (speech, text, and facial analysis) into a seamless user experience.
• Building a scalable app that allows users to practice interviews from any device, using just a microphone and webcam.
• Implementing real-time feedback with detailed insights on speaking style, body language, and interview performance.
• Delivering a smooth and engaging user experience despite the complexity of integrating speech and facial expression analysis.

What we learned

• AI Integration: Working with AI models (speech-to-text, text-to-speech, facial recognition) taught us how to efficiently integrate and optimize multiple AI services into a single system.
• Frontend and Backend Communication: Managing data flow between the frontend and backend was a learning curve, particularly for handling real-time audio and video data.
• User Experience Design: Building an intui

Built With

• fastapi
• openai
• python
• typescript