Inspiration
Refugees who don't speak English have huge problems navigating the asylum system. Many do not have access to a computer - particularly those in detention.
What it does
Karibu is a phone-based legal assistant that can take incoming calls and understands and speaks many languages. It is connected to a database of legal advice.
Karibu can understand
Afrikaans, Arabic, Armenian, Azerbaijani, Belarusian, Bosnian, Bulgarian,
Catalan, Chinese, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish,
French, Galician, German, Greek, Hebrew, Hindi, Hungarian, Icelandic,
Indonesian, Italian, Japanese, Kannada, Kazakh, Korean, Latvian, Lithuanian,
Macedonian, Malay, Marathi, Maori, Nepali, Norwegian, Persian, Polish,
Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swahili,
Swedish, Tagalog, Tamil, Thai, Turkish, Ukrainian, Urdu, Vietnamese, Welsh
and can speak
Afrikaans, Arabic, Bulgarian, Catalan, Chinese, Czech, Danish, Dutch, English,
Finnish, French, Galician, German, Greek, Hebrew, Hindi, Hungarian, Icelandic
Indonesian, Italian, Japanese, Korean, Latvian, Lithuanian, Malay, Marathi,
Norwegian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Spanish,
Swedish, Tamil, Thai, Turkish, Ukrainian, Vietnamese, Welsh
in a native voice.
How we built it
Karibu is implemented using Twilio, Anthropic Claude, MongoDB Atlas Vector Search, Whisper and GCP's text-to-speech.
Challenges we ran into
Twilio's voice transcription is inaccurate and requires the language to be specified in advance. Waiting for Twilio to provide recordings is also slow. By listening in to the call, using websockets, and transcribing it ourselves, we can support multilingual transcriptions, improve accuracy and reduce the latency by ~5 seconds.
Accomplishments that we're proud of
Building something that can take a phone call, transcribe it in many languages, research an answer from a high quality dataset (in this case, .gov.uk) and speak back to the user in their native language in <24 hours!
Log in or sign up for Devpost to join the conversation.