Fluently

Fluently transforms speech practice into proven skill growth by analyzing your delivery, diagnosing real-time weaknesses, and providing targeted adaptive drills to help you speak with confidence

Comment

Inspiration

Public speaking, interviewing, and spontaneous communication are among the most critical skills for personal and professional growth, yet traditional public speaking apps suffer from a fundamental flaw: they give scores without showing you how to actually improve. Most existing tools present a single arbitrary percentage or dump generic LLM summaries after a recording. We wanted to build something different: a system that moves away from one-off "evaluations" and functions like a real human speech coach. Fluently was born from the vision of an adaptive practice loop—Practice → Diagnose → Train → Retry → Compare → Improve—where every piece of feedback is grounded in hard acoustic and linguistic data, not decorative AI text.

What it does

Fluently was architected around a strong separation of concerns between frontend experience, signal processing, and cognitive AI reasoning: Frontend: Built with React, TypeScript, Vite, and Tailwind CSS. It leverages the browser Web Audio API (MediaRecorder) for gain-normalized audio recording, chunking, and real-time visualization. Transcription Stage: Uses OpenAI Whisper for high-accuracy speech-to-text with word-level timestamps. Acoustic Processing Engine: Built with Python & Librosa to perform spectral audio analysis, pitch tracking, and speech pause detection. NLP & Text Engine: Uses spaCy for syntactic parsing, filler word extraction, and vocabulary complexity measurement. AI Cognitive Layer: Powered by Anthropic Claude, acting as the reasoning conductor that translates deterministic metrics into actionable human coaching and dynamic practice prompts. Backend API & Data Persistence: Powered by FastAPI with asynchronous pipeline orchestration, backed by Supabase (PostgreSQL) for user profiles, session state, and metric provenance.

How we built it

Fluently combines a modern, responsive web application with a multi-stage Python analytical engine: Frontend & Audio Recording: Built using React, TypeScript, Vite, and Tailwind CSS. It uses the browser's Web Audio API (MediaRecorder) for real-time audio chunking, gain normalization, and dynamic waveform visualization. Transcription Stage: Powered by OpenAI Whisper, extracting word-level timestamps, transcripts, and pause boundaries. Deterministic Acoustic Engine: Uses Librosa to measure pitch variation, speaking rate (WPM), articulation speed, and acoustic energy distribution. Deterministic NLP Engine: Uses spaCy for lexical diversity (type-token ratio), filler word density, grammatical coherence, and sentence structural complexity. AI Cognitive & Coaching Engine: Powered by Anthropic Claude, which consumes the raw deterministic metrics from Librosa and spaCy to generate personalized coaching reports, explain root weaknesses, and generate target practice drills. Data & Profile Persistence: Supabase (PostgreSQL) stores longitudinal skill profiles, session history, and metric provenance so users can track real skill progression over time.

Challenges we ran into

  1. Eliminating "False Precision" & Separating Measurement from LLM Reasoning The Challenge: Delegating speech scoring entirely to an LLM creates significant issues—models hallucinate metrics, give inconsistent scores for identical audio, and blur objective measurements with subjective commentary. The Solution: We architected a strict boundary: software measures; AI interprets. We separated signal processing from cognitive reasoning. Librosa and spaCy compute 100% deterministic, mathematically sound metrics (e.g., exact filler ratio, pitch variance in Hz, WPM). Claude receives these locked empirical data points as ground truth, using them strictly to synthesize actionable feedback and custom practice drills without inventing numbers.
  2. Managing Multi-Stage Async Latency & CPU-Bound Bottlenecks The Challenge: Running sequential heavy tasks—Whisper transcription, Librosa pitch tracking, spaCy NLP tokenization, and Claude API calls—resulted in unacceptable user waiting times. Librosa and spaCy are CPU-heavy Python routines that blocked the main event loop in FastAPI. The Solution: We built a custom Async Pipeline Orchestrator using Python’s asyncio and thread pool executors. Once Whisper finishes transcription, the pipeline executes the acoustic signal analysis (Librosa) and linguistic parsing (spaCy) concurrently in parallel worker threads, reducing total analysis pipeline latency by ~50%.
  3. Cross-Browser Web Audio Inconsistencies & Audio Artifacts The Challenge: Different browsers encode audio streams differently (Opus in WebM vs. AAC in MP4), often introducing variable sampling rates, background hum, or initial silent buffers. These artifacts corrupt Librosa pitch contour detection and cause Whisper to hallucinate timestamps. The Solution: We implemented client-side audio preprocessing using Web Audio API gain nodes, combined with backend tempfile standardisation and dynamic energy-threshold silence filtering before passing buffers into the signal processing modules.

Accomplishments that we're proud of

Zero-Hallucination Metric Engine: Built a speech application where every score and chart is 100% verifiable against real audio frequencies and transcript tokens. Instant Actionable Retry Loop: Designed an intuitive UX where users can immediately click "Retry with Drill" right from their diagnostic screen to put feedback into practice instantly. High-Performance Async Pipeline: Successfully orchestrated four distinct AI/DSP libraries (Whisper, Librosa, spaCy, Claude) into a cohesive sub-second backend pipeline. Longitudinal Progress Tracking: Created a persistent skill profile system that proves tangible user progress over weeks of practice.

What we learned

Hybrid Architectures Excel: Combining deterministic Python data tools (DSP/NLP) with Generative AI produces vastly more reliable, trustworthy, and scalable applications than relying on prompt engineering alone. Speech Mechanics are Multi-Dimensional: Pace, pitch, pause distribution, and filler words interact dynamically; measuring pitch variation alongside cadence is essential for assessing true speaking confidence. UX Drives Habit Formation: The key to skill acquisition is reducing friction between receiving feedback and retrying the task.

What's next for Fluently

Domain-Specific Scenarios: Expanding practice modes for tech interviews, investor elevator pitches, academic presentations, and group discussions (GDs). Real-Time Live Coaching: Adding in-session visual cues (such as a live pace indicator) to alert speakers when they are rushing or overusing filler words in real time. Multi-Speaker Group Analysis: Extending the analysis pipeline to process multi-person audio recordings for meeting dynamics and conversation turn-taking analysis. Mobile Native App: Bringing Fluently to iOS and Android for on-the-go quick practice sessions before important meetings or speeches.

Built With

Share this project:

Updates

Submission history