Hey Buddy

Language shouldn't cut anyone off. Buddy joins any phone call to interpret both sides live and warn people about scams, and gives energy and health hotlines a record of every call.

Comment

Inspiration

Many of us have family members who understand English but freeze on the phone: a pharmacist talking fast, a clinic confirming a time, a utility asking for an address. The same people are the ones phone scammers target, with a "grandson" in jail who needs bail and your bank account information, warning you not to tell anyone. The family usually hears about it afterwards.

It is the same problem on the other end of the line. A gas utility, a community clinic or a plant's safety hotline gets calls from people who can describe a leak or a symptom perfectly well, just not in English. Those are the calls where a misunderstanding is most expensive.

What it does

For people: Buddy is an interpreter you add to the phone call you are already on. There is no app to install:

  1. During any call, tap Add call, dial Buddy's number, and tap Merge.
  2. Buddy greets both sides, then interprets turn by turn: the other person hears you in English, and you hear them in your language. It works in English, Turkish, Spanish, Arabic, Hindi (beta) and Mandarin (beta).
  3. If the caller starts using scam tactics, such as pressure to pay in gift cards or wire transfers, a demand for secrecy, a request for a code or password, or asking you to install an app, Buddy warns you in your own language right after its interpretation, and sends the same warning on WhatsApp. It quotes the caller's exact words; it never says a call is "safe".
  4. Afterwards, the portal (phone number + PIN) shows every call as a conversation thread with the original and the interpretation side by side, the audio of each turn, and a short summary written by Meta Muse Spark. The summary and the warnings are in the user's language, with English versions available on request.

For businesses and hotlines: every line (an ad campaign, a website number, a safety hotline) gets its own tracking number. Each interpreted call becomes a CRM record automatically: where the caller came from, what they wanted (booking, pricing, safety report, investor inquiry, …), the outcome, follow-ups, and the full bilingual transcript. Urgent safety calls, like a resident reporting a gas smell in Spanish, are marked escalated. The dashboard shows calls by line, language and intent. The whole web app, including the CRM, works in the same six languages, with right-to-left layout for Arabic.

How we built it

  • Telephony: a Twilio number whose webhook opens a Media Streams WebSocket. We segment it into utterances on the server, and Buddy's voice goes back into the same call.
  • Speech-to-text: ElevenLabs Scribe v2 with automatic language detection plus per-language key terms (so "torunum", "kefalet" or "gift card" are not misheard). The spoken language decides who is talking and in which direction to interpret. A low-confidence result is retried once in the user's language.
  • Translation and scam cues: xAI Grok (grok-4.20 non-reasoning) translates each turn with a glossary for scam-relevant terms, and separately reads the other party's recent turns for six manipulation cues. A warning is spoken only for a strong cue (payment, secrecy, credentials, remote access), at most once per new cue per call. "This is the police" or "your grandson is in trouble" on its own is recorded for the portal but never interrupts the call.
  • Speech: ElevenLabs Flash v2.5 with a native voice per language, straight into the call. The greetings for all six languages are pre-synthesized at startup.
  • Post-call summary: Meta Muse Spark writes two or three sentences in the user's language after the call ends.
  • App: Node.js + Express + Socket.IO server and a React + Vite + Tailwind web app, served from one process on Railway, with a persistent volume for call history and audio, on hey-buddy.tech.
  • Safety by construction: the caller's words are always treated as data, never as instructions, so "ignore the warnings" gets interpreted rather than obeyed. Portal and CRM logins are rate-limited. Each business key only sees its own tracking numbers. A tests-only audit pass found and fixed ways a malformed stream frame or login attempt could crash the server.

Challenges we ran into

  • Latency on a real phone line. Each turn is STT → translation → speech, and every second of silence feels long on a call. We started with a translation model that spiked to 3–13 s on phone audio and moved translation to Grok, which answers in about 0.6–0.8 s. The pipeline now takes 1.1–1.4 s per turn on a real merged call; from the end of speech until Buddy starts speaking, the median on real calls was 1.4–1.8 s.
  • Warnings that arrive in time. We first ran scam analysis on Meta Muse Spark; on a live call it hit our 8-second budget and the warning never played. We moved live analysis to Grok (1.3–1.5 s, in parallel with the interpretation) and gave Muse the post-call summary, where a few seconds don't matter.
  • Who is speaking? A merged three-way call mixes everyone into one audio stream. We use the detected language plus the user's saved language to decide who spoke and which way to interpret.
  • Merging calls. Buddy used to greet the instant it answered, which collided with the carrier merging the calls and garbled the greeting. It now waits for the merge.
  • Warnings that are useful, not noisy. A lone "this is the police" is suspicious but common. Our first policy warned on it, so we moved to warning only when a demand follows: payment, secrecy, a code or remote access.
  • Carrier rules. SMS from a new local number was filtered until A2P registration, so warnings go out over WhatsApp.

Accomplishments that we're proud of

  • A working interpreter on ordinary phone calls, with no app and no special hardware: verified on real calls in Turkish, with Spanish, Arabic, Hindi and Mandarin tested end to end on phone-quality audio.
  • Scam warnings that show their evidence (the caller's exact words) instead of asking to be trusted.
  • One product for both sides of the language gap: the person calling and the business or hotline answering.
  • The whole experience, including the business CRM, available in six languages.

What we learned

  • For live interpretation, the response time of the translation model matters more than anything else.
  • Put each model where its latency fits: the fast ones inside the call, the slower ones after it.
  • Phone audio (8 kHz, compressed, three people on one line) is much harder than a laptop microphone. Key-term hints and a single retry in the user's language made the biggest difference.
  • Safety features have to be conservative on purpose: quoting evidence and never saying "this call is safe" is what makes them trustworthy.

What's next

  • Native-speaker review of the Arabic, Hindi and Mandarin experience, and testing with older users and community organizations.
  • Registered SMS alerts, and letting the user pick a trusted family member to receive the warning.
  • Pilots with a utility or clinic hotline, starting with safety-critical calls.

Meta challenge: Bringing People Closer Together with AI

Who it is for. People in the US who are not comfortable on English phone calls, often older immigrants, and the families who worry about them. On the other end are the clinics, utilities and other businesses that want to understand them.

How it strengthens human connection. Buddy does not replace the conversation; it lets two people actually have it. The user speaks for themselves, in their own voice and language, instead of handing the phone to a relative or hanging up confused. When something sounds wrong, the warning arrives during the call, when it can still change what happens, and it sends the user back to the people they trust: in our demo, the grandparent's answer is "I'll call my son first." The portal lets the family look back at what was said together.

Why AI is necessary. Recognizing speech from a short, noisy 8 kHz phone clip in six languages, translating it in under a second while keeping names, amounts and negations intact, and telling the difference between "buy gift cards and read me the numbers" and "never give anyone a gift-card code": none of this works with rules or keyword lists. The models do the language work. The product decides when to speak up, and always shows the exact words that triggered it.

Built With

Share this project:

Updates

Submission history