• 

  Dashboard

• 

  Live working

• 

  Dashboard

• 

  Files used

• 

  Code

• 

  Code

• 

  Live testing

• 

  You Tube Video

Inspiration :

During mock interviews and presentations, communication anxiety often causes fast pacing, awkward pauses, and excessive filler words. Traditional practice lacks measurable, objective feedback. We built SpeakTwin-AI as an intelligent speech mirror that delivers real-time analytics on rhythm, pauses, and clarity to help speakers build genuine confidence.

What it does :

AudioPulse is a speech-coaching and dataset-studio application for candidates preparing for internationaltechnical interviews. It listens to a spoken answer (live microphone or uploaded audio), transcribes it,locates errors in grammar, word choice, fillers and pacing, rewrites the sentence in polished executivephrasing, and produces transparent scores. A built-in Custom Speech Dataset module (the SpeakTwin AIResearch Corpus) turns every practice session into paired, labelled training data. The speech analysisruns on an onboard algorithmic engine (rules and signal processing), so scores are deterministic,explainable and reproducible; an optional "Deep AI Diagnostic" adds a model-based second opinion.

How we built it :

Frontend Architecture: React with TypeScript, bundled via Vite for fast load times and clean component modularity.
Speech AI Engine: Google AI Studio / Gemini API integration for natural language evaluation, deep diagnostics, and contextual rewriting.
Audio & Signal Processing: Web Audio API and onboard signal-processing algorithms for deterministic audio analysis (pitch, silence thresholds, cadence). Styling & UI: Modern dark-mode dashboard tailored for real-time analytics visualization (metric cards, progress bars, interactive audit panels). Version Control & Collaboration: Git, GitHub, feature branching, and pull requests

Challenges we ran into

• Balancing Real-Time Audio with AI Latency: Running heavy AI models caused lag; we resolved this with a hybrid model—using fast, client-side algorithms for instant metrics (WPM, pauses) and async API calls for deep diagnostics.
• Handling Natural Pauses vs. Hesitations: Tuning audio thresholds (>1.2s) to tell the difference between intentional pauses for emphasis and nervous hesitation stalls.
• Browser Audio & Cross-Platform Codecs: Handling varied microphone permissions and normalizing differing audio formats across browsers in React.
• Team Collaboration & Merge Conflicts: Coordinating fast-paced code changes across feature branches without breaking dependencies or UI states.

Accomplishments that we're proud of

Built a Fully Functional Speech Dashboard: Successfully created and integrated an interactive analytics interface that provides immediate feedback on pacing, hesitation stalls, and filler words. Hybrid Dual-Engine Architecture: Successfully combined an onboard, deterministic signal-processing engine for zero-latency metric calculations with deep AI diagnostic capabilities. Executive Phrasing & Rewriting: Integrated dynamic sentence transformation that elevates informal, filler-heavy candidate responses into concise, executive-level communication. Integrated Research Corpus Pipeline: Automated the creation of paired, labeled training data from live practice sessions to support continuous speech research. Seamless Collaborative Delivery: Managed rapid branch integration, clean Git workflows, and deployment under tight hackathon timelines.

What we learned

Web Audio API & Real-Time Buffer Processing: Learned how to tap into AudioContext and AnalyserNode to sample time-domain data, extract real-time decibel levels, and calculate speech silence thresholds programmatically. Deterministic Signal-Processing Algorithms: Implemented algorithmic heuristics in TypeScript/JavaScript to compute Words Per Minute (WPM) dynamically from timestamped transcription tokens without depending on cloud-based LLM latency. Asynchronous LLM Orchestration: Mastered integrating the Gemini API asynchronously using structured JSON schemas, ensuring speech rewrites and diagnostic evaluations stream back without blocking client-side UI threads. React State & Stream Lifecycle Management: Gained experience managing complex React hook lifecycles (useEffect, useRef) for persistent media streams, safely stopping mic hardware tracks, and preventing memory leaks during audio recording sessions. Type-Safe Full-Stack Architecture: Enforced strict TypeScript interfaces across the app from raw audio payload representations to AI diagnostic response reducing runtime errors during live demonstrations.

What's next for SpeakTwin-AI

Real-Time Conversational AI Mock Interviewer: Upgrade from static audio reviews to an interactive, full-duplex conversational agent that conducts live mock technical interviews and asks dynamic follow-up questions. Multi-Modal Visual & Body Language Analysis: Integrate WebRTC video tracking to monitor eye contact, facial expressions, posture, and nervous gestures alongside vocal metrics. Expanding the SpeakTwin AI Research Corpus: Open-source an anonymized benchmark dataset of paired technical interview responses to support community research on speech intelligibility and non-native speaker accents. Personalized Accent-Adaptive Feedback: Develop custom coaching profiles that tailor pacing and intonation recommendations to individual non-native English speakers without forcing them to erase their authentic cultural accents. Seamless Platform Integrations: Build browser extensions and plugins for platforms like Zoom, Google Meet, and LeetCode to provide real-time speech analytics and feedback during practice coding sessions.

Built With

• css3
• git
• github
• html5
• javascript
• python
• react
• typescript
• vite
• web-audio-api