PitchPerfect Ai

AI Sales Role Play Platform Built to Win Every Call

Comment

Here is the complete, polished Devpost submission for PitchPerfect AI.

I have structured this exactly how hackathon judges love to read it: starting with a relatable problem, diving deep into the Google tech stack, explaining the clever prompt engineering we reverse-engineered, and highlighting the math behind the grading engine.

Inspiration

B2B sales training is fundamentally broken. It is expensive, time-consuming, and usually relies on static scripts or awkward roleplay sessions with a manager. While early "AI voice bots" entered the market recently, they suffer from two fatal flaws: high latency (making conversations feel like talking walkie-talkies) and blindness (they can't see the slide decks or software demos that drive modern B2B sales).

When Google announced the Gemini Multimodal Live API, we realized we finally had the technology to build the holy grail of sales enablement: an AI prospect that can hear your pitch with ultra-low latency, interrupt you when you talk too much, and actually look at the screen you are sharing to critique your presentation. We built PitchPerfect AI to democratize enterprise-grade sales coaching.

What it does

PitchPerfect AI is a real-time, multimodal sales roleplay platform.

  1. The Setup: A sales rep selects a target buyer persona (e.g., "David Williams, a busy 42-year-old VP of Sales") and selects dynamic objections they want to practice (e.g., "Budget cuts" or "Tool fatigue").
  2. The Simulation: The rep starts the call, sharing their microphone and screen. The Gemini-powered AI prospect speaks in real-time. If the rep shares a pricing slide, the AI can literally "see" it and ask, "Why is the enterprise tier so much higher than what I'm paying now?"
  3. The Autopsy: The moment the call ends, PitchPerfect aggregates the transcript and generates a brutal, highly accurate coaching scorecard. It grades the rep across 5 core pillars (Opener, Discovery, Social Proof, Takeaway, Closing) and quotes exact timestamps from the transcript to provide actionable feedback.

How we built it

To achieve sub-second conversational latency, we completely bypassed the traditional, clunky STT $\rightarrow$ LLM $\rightarrow$ TTS pipeline.

  • The Live Engine: We built a Next.js frontend that captures raw PCM audio from the user's microphone and extracts Base64 JPEG frames from their screen share (~1 FPS). These are streamed directly over WebSockets to the Gemini Multimodal Live API (gemini-2.0-flash-exp). Because the model natively processes and returns audio waveforms, the conversation feels incredibly human.
  • The "Brain" (Prompt Engineering): We engineered massive, state-machine-like systemInstructions. We gave the AI a strict identity, company details, and behavioral constraints (e.g., "You are forbidden from being helpful. You must interrupt if they monologue.").
  • The Grading Engine: Post-call, we use Gemini 1.5 Pro with Structured Outputs (JSON mode). We pass the raw transcript into a custom prompt that forces Gemini to evaluate the rep against a strict commercial sales rubric.
  • The Scoring Math: The final call score isn't an arbitrary LLM guess; it is deterministically calculated on the backend based on the AI's criteria validation:

$$Score_{final} = \left( \frac{\sum_{i=1}^{n} (w_i \cdot p_i)}{\sum_{i=1}^{n} w_i} \right) \times 100$$

(Where $w_i$ is the weight of the specific sales criterion, such as "Permission-based opener," and $p_i \in {0, 1}$ represents whether the AI determined the rep passed that check).

Challenges we ran into

  • Taming the AI: LLMs are inherently designed to be helpful and compliant. Forcing Gemini to act like a grumpy, dismissive executive who wants to hang up the phone required intense negative prompt engineering. We had to specifically instruct it to withhold information so the human would be forced to ask good discovery questions.
  • Streaming Binary Data: Handling raw Web Audio API PCM streams and syncing them with visual screen-capture frames over a single WebSocket connection without overloading the browser's memory was a massive frontend hurdle.
  • Deterministic Grading: Initially, asking an LLM to "grade the call" resulted in vague advice. We had to reverse-engineer commercial grading rubrics to force Gemini 1.5 Pro to output a strict JSON schema that graded specific, binary actions (e.g., "Did the rep re-confirm the calendar invite time? True/False").

Accomplishments that we're proud of

  • True Multimodality: We successfully moved beyond a standard "voice bot." When a user shares their screen showing a complex dashboard, the AI actually understands the UI and incorporates it into its objections.
  • Zero-Latency Feel: By leveraging Gemini's native audio-in/audio-out capabilities, we achieved response times that allow for natural human interruptions and overlapping speech.
  • Enterprise-Grade Architecture: We didn't rely on third-party voice wrappers. We built directly on Google Cloud and the Gemini SDK, proving this architecture is scalable and production-ready.

What we learned

  • The Power of Native Audio: We learned exactly how much overhead is wasted in traditional Voice AI pipelines. Bypassing text translation entirely changes the paradigm of human-computer interaction.
  • Advanced Structured Outputs: We mastered using Gemini's JSON mode to guarantee perfectly typed API responses, allowing us to pipe complex AI evaluations directly into our React UI components without parsing errors.
  • WebRTC and WebSockets: We gained a deep, hands-on understanding of how to manage binary streams and audio contexts in modern web browsers.

What's next for PitchPerfect AI

  • Real-Time Coaching Nudges: We plan to multi-thread the analysis so the UI can flash live warnings (e.g., "You are talking too fast!" or "Ask an open-ended question!") during the call.
  • CRM Integrations: Connecting the platform to Salesforce or HubSpot so reps can click one button to instantly generate an AI roleplay persona based on a real, live prospect in their pipeline.
  • Voice Cloning: Integrating custom voice profiles so reps can practice against the specific accents and tones of their target demographics.

Built With

  • application-and-api-development-framework
  • audio-capture-support/fallbacks-ai-realtime-api
  • client-side-transcript/session-persistence-deployment-platform
  • component-based-ui-architecture-styling
  • container-based
  • deployment
  • for
  • frontend-and-backend-api-routes-ui
  • google-cloud-run
  • google-gemini-api-(gemini-2.0-flash-/-1.5-pro-style-structured-output)
  • google-gemini-live-api-(gemini-2.0-flash-exp)
  • hackathon
  • json-+-structured-output-schema
  • low-latency-bidirectional-voice-streaming-browser-audio
  • mediarecorder-api
  • mic-capture-and-pcm-processing/playback-browser-recording
  • next.js-(app-router)
  • node.js
  • persona-data-and-scorecard-contract-state-persistence
  • post-call-coaching-scorecard-generation-data-format
  • prebuilt-accessible-ui-primitives-runtime
  • react
  • real-time-voice-roleplay-agent-ai-scoring-api
  • server/runtime-for-next.js-realtime-transport
  • sessionstorage
  • shadcn/ui
  • tailwind-css
  • target
  • typescript
  • utility-first-styling-ui-components
  • vercel
  • web-app-hosting-and-ci-deployment-cloud-platform-(target)
  • web-audio-api
  • websocket-api
Share this project:

Updates

Submission history