it'sPEAK

Home page
Project page
Session page, with video and audio analysis report
Session page, with light mode

Inspiration

Public speaking is an important skill, yet university students and early-career professionals do not always have access to timely, personalised coaching. Practising alone can also make it difficult to identify specific habits or measure improvement objectively. We created it'sPEAK to give users a private space to rehearse presentations, interviews, pitches, and speeches while receiving actionable feedback grounded in observable delivery.

What it does

it'sPEAK is an AI-powered public-speaking coach. Users create rehearsal projects, choose the areas they want to improve, select a presentation archetype, and upload an English-language video of up to three minutes.

Before analysis begins, the application checks the recording’s video, audio, lighting, framing, face visibility, and duration. This helps users correct technical problems that could otherwise make the feedback unreliable.

The vocal-analysis pipeline measures pacing, intonation, filler-word usage, and pauses. Visual analysis evaluates observable signals such as eye contact, facial expression, posture, gestures, movement, and use of space.

Rather than applying one universal standard, it'sPEAK calibrates its scoring to different speaking contexts or "archetypes". These include: corporate presentation, academic conference, motivational keynote, startup pitch, informal team presentation, and job interview, which can therefore have different expectations for expression, movement, and delivery.

After analysis, users receive a skill breakdown, progress charts, a synchronised video player, with landmark overlays and an eye-contact timeline, prioritised coaching cards, and an editable transcript. Each project retains up to five successful rehearsals, with the first session protected as a baseline for measuring progress.

How we built it

We built the frontend with Next.js, React, and Tailwind CSS. Clerk manages user authentication and provides session tokens for protected API requests, while FastAPI exposes the backend endpoints for projects, uploads, reports, transcripts, and private artifacts.

Uploaded recordings are inspected with FFmpeg before analysis jobs are placed on a Celery queue backed by Redis. MediaPipe extracts facial and body landmarks, while Librosa’s YIN pitch tracker and supporting audio metrics analyse vocal delivery. OpenAI Whisper generates word-level transcripts, and an OpenAI language model produces grounded coaching. If coaching generation is unavailable, the system falls back to deterministic feedback based on the measured scores.

Supabase stores projects, session metadata, reports, videos, and landmark artifacts. Access to application data is owned and enforced by the backend, while private videos and analysis artifacts are delivered through short-lived signed URLs.

We deployed the frontend on Vercel and the containerised backend on Railway. The Railway service runs FastAPI, Celery, and the cleanup scheduler alongside managed Redis and persistent temporary storage. To operate within limited memory, the analysis worker processes one recording at a time and restarts cleanly after every completed analysis.

Challenges we ran into

One major challenge was handling incomplete pose and face tracking. Early versions risked interpreting missing landmarks as poor performance, which could unfairly reduce a user’s score. We changed the pipeline so missing measurements are represented as insufficient or low-confidence data rather than automatically becoming zero.

We also had to manage uploaded videos, background processing, private storage, and retained rehearsal sessions safely. A session should only replace an older rehearsal after the new analysis succeeds, and the first successful session must remain protected as the project baseline. This required coordinating Redis jobs, temporary files, Supabase transactions, and artifact cleanup.

Developing fair scoring expectations was another challenge. Different presentation styles call for different levels of expression, movement, pacing, and formality. We introduced presentation archetypes so identical behaviour can be evaluated according to its intended context rather than against a single generic standard.

Audio analysis also required significant optimisation and calibration. We replaced a slower pitch-tracking approach with Librosa’s YIN algorithm and normalised pitch variation relative to each speaker’s median pitch. We also refined our pacing and intonation targets to better reflect Singaporean English delivery and tested transcription with familiar local expressions such as “lah” and “walao.”

Finally, running MediaPipe and Librosa together created substantial memory pressure in production. We reduced duplicate imports, released visual-analysis data before starting audio analysis, cached compiled audio kernels, and recycled the worker between recordings. These changes reduced analysis time and allowed consecutive uploads to complete reliably.

Accomplishments that we're proud of

it'sPEAK combines vocal, facial, and body-language analysis into one coherent rehearsal experience rather than presenting disconnected technical measurements. Users can move from recording quality checks to analysis, coaching, video evidence, and progress tracking within a single application.

We also implemented private authentication, durable database persistence, protected artifact storage, asynchronous video processing, and a five-session rehearsal lifecycle. The first session acts as a stable baseline, while later sessions help users see whether specific delivery skills are improving, stagnating, or declining.

Most importantly, we designed the feedback to remain grounded in observable behaviour. The application focuses on what was measured in the recording and avoids making unsupported claims about a user’s emotions, personality, anxiety, or employability.

What we learned

We learned that missing data should never be confused with poor performance. Computer-vision confidence can change because of lighting, framing, occlusion, or camera quality, so a responsible system must clearly distinguish between a weak measurement and a weak delivery.

We also learned that public speaking cannot be evaluated using one universal ideal. A motivational keynote may benefit from expressive movement, while a job interview may call for a more controlled delivery. Scoring becomes more useful when it reflects the speaker’s intended context.

Another important lesson was that generative AI is most valuable when it is constrained by reliable measurements. Instead of asking AI to judge an entire performance freely, we provide it with structured scores, observable metrics, and selected improvement areas. Deterministic fallbacks ensure that coaching remains available even when generation fails.

Finally, we learned that production machine-learning pipelines require different engineering decisions from local prototypes. Memory usage, task retries, persistent files, worker lifecycle, and processing visibility became just as important as model accuracy.

What's next for it'sPEAK

We plan to support longer recordings and additional languages, including bilingual presentations. This will require language-aware transcription, filler-word detection, pacing targets, and coaching that can handle speakers switching naturally between languages.

We also want to introduce more specialised presentation archetypes for settings such as sales demonstrations, panel discussions, and investor updates. In the longer term, users could define custom archetypes, with AI helping translate their goals and audience expectations into transparent scoring criteria.

Progress tracking can also become richer. Future versions could identify recurring habits across sessions, recommend targeted exercises, highlight each user’s strongest rehearsal, and provide more detailed comparisons between recent sessions and the original baseline.

As the user base grows, we will need to scale the analysis pipeline beyond a single worker, move pending uploads into shared object storage, and distribute processing across multiple services. We also plan to continue validating the scoring system across more speakers, accents, recording environments, and presentation contexts.

Frontend Key Features

The frontend organises rehearsals into private project folders. Users can define a goal and deadline, select a speaking archetype, and choose specific vocal or visual improvement areas. Each project presents its retained sessions, baseline status, latest results, and progress over time in one focused workspace.

The upload experience guides users through recording selection, quality checks, warning confirmation, background processing, and session replacement. Progress is displayed without regressing between polling updates, while clear recovery states explain when users should confirm a warning, replace an older session, or record a new video.

Completed reports combine an overall score with separate vocal, facial, and body-language ratings. Selected improvement areas are prioritised by score, while proficient areas remain visible without generating unnecessary criticism. Radar charts and progress timelines help users understand both the current rehearsal and changes across sessions.

The synchronised video-analysis player loads private videos and landmarks through signed URLs. Users can review face and pose overlays while the recording plays and navigate directly to sections of the eye-contact timeline. Reports also include an editable transcript so users can correct transcription errors without repeating the full analysis.

The interface supports responsive layouts, keyboard-accessible controls, clear loading and error states, and light and dark themes. Clerk protects project and session routes, while the browser requests all application data through the authenticated backend rather than accessing the database directly.

Backend Technical Features

The FastAPI backend provides authenticated endpoints for project management, session uploads, quality confirmation, status polling, transcript correction, archetype discovery, and private artifact access. Clerk session tokens establish the user identity, and every project, session, report, and artifact request is checked against that ownership boundary.

Video analysis runs asynchronously through Celery and Redis. A lightweight quality-check task validates the recording before the more computationally intensive analysis begins. Late task acknowledgements and lost-worker handling allow interrupted jobs to return to the queue instead of disappearing silently.

The visual pipeline uses MediaPipe landmarks to derive eye-contact, expression, posture, gesture, movement, and spatial-use measurements. The audio pipeline uses FFmpeg for extraction, Librosa YIN for pitch tracking, and OpenAI Whisper for word-level transcription. Deterministic scoring converts the raw measurements into bounded, archetype-aware scores before coaching is generated.

The coaching layer sends structured measurements, selected improvement areas, and presentation context to OpenAI. Generated feedback is validated before being shown to the user, and rule-based fallbacks keep the pipeline functional if the model request fails or returns unusable output.

Supabase provides durable project and session storage, transactional five-session retention, private video and landmark storage, and short-lived signed artifact URLs. Session 1 is protected as the baseline, failed analyses do not consume a retained-session slot, and replacement occurs only after a new analysis has completed successfully.

The production backend is packaged as a Docker container containing FastAPI, a solo Celery worker, Celery Beat, FFmpeg, MediaPipe, and Librosa. Production preflight checks reject missing credentials, local-only service URLs, and unwritable storage. Memory-aware cleanup, persistent Librosa caches, and automatic worker recycling allow repeated analyses to run reliably on constrained cloud infrastructure.