BabelDesk

AI that untangles chaotic multilingual conversations — separating speakers, understanding every accent, and extracting structured actionable intelligence in real-time.

Comment

Inspiration

Multilingual meetings across Southeast Asia are chaotic — multiple accents, code-switching, people talking over each other. Critical decisions get lost, action items are missed, and some voices are never heard. We asked: what if AI could untangle the chaos?

What it does

BabelDesk takes chaotic multi-speaker audio and produces structured, actionable intelligence. It separates individual speakers, transcribes each with accent-aware understanding (Singlish, Vietnamese, Thai, etc.), analyzes sentiment per speaker, and extracts decisions, action items, risks, conflicts, and even detects when someone was talked over.

How we built it

  • Speaker Diarization: pyannote-audio identifies "who spoke when," even with overlapping speech
  • Accent-Aware Transcription: VALSEA API handles real-world SEA accents and code-switching — going beyond basic transcription to extract semantic meaning
  • Intelligence Extraction: OpenAI GPT-4o converts the speaker-labeled transcript into structured JSON (decisions, action items, risks, verbal approvals, conflicts)
  • Backend: FastAPI orchestrating an async pipeline
  • Frontend: Real-time dashboard showing speaker profiles, sentiment, and structured meeting output

Challenges we ran into

Handling overlapping speech where multiple speakers talk simultaneously was the hardest part. We also had to carefully design our pipeline to use VALSEA's full API surface — not just transcription, but annotation, sentiment, clarification, and translation endpoints — to go beyond what basic STT can do.

What we learned

Acoustic features like speaker voice signatures are language-agnostic — the same diarization model works regardless of what language is spoken. This makes the approach scalable across all of Southeast Asia without retraining.

What's next

Real-time streaming mode, integration with Slack/Teams for automatic post-meeting reports, and a "meeting equity" score that tracks whether all participants get fair speaking time.

Built With

Share this project:

Updates