SpeaKore

Speak any language, out loud and live, and let Kore run your apps by voice. Real-time interpreting and agentic actions, powered by NVIDIA Nemotron on Nebius Token Factory.

Comment

Inspiration

I live in Spain, and the hardest part isn't the paperwork. It's the people. Sure, calling the town hall, the tax office or a doctor in a language you only half speak is stressful, and the stakes are real. But what I miss most is the easy stuff: asking someone the way, joining the conversation at a dinner table, laughing at the joke, playing cards with the neighbours.

When a table is full of people from different countries, you usually fall back on English. It works, but it is never the same as speaking your own language. You think a little slower, you joke a little less, and the quietest person at the table says the least.

Translation apps don't fix that. They make you type, wait and pass your phone around, and by the time the answer arrives the moment is gone. I wanted something that simply interprets out loud, live, so everyone can talk in their own language. And because life is also full of small chores, I wanted it to do things for you too: "email the bank that my card must be blocked", "what's on my calendar tomorrow?".

What it does

SpeaKore is a live voice interpreter plus a hands-free assistant called Kore. Behind the scenes there are two pipelines that never mix: translate and control. Translating can never trigger an action.

  • At the table: one phone in the middle, 2 to 8 people, each in their own language, for a dinner, a card night or a meeting. Tap your seat and speak, and every other seat gets your words in their language, dealt like playing cards around a poker table. Tap a card to hear it read aloud.
  • In the street: one phone between two people, each with their own talk button. Ask the way, and the answer comes back in your own language, spoken right away.
  • Kore, voice control: "go to connectors", "switch to dark mode", "read my latest mail", "send Maria an email that I'm late". Reading happens instantly. Anything that sends or changes something always shows a confirmation card first. Kore never moves money.
  • Connectors: just give Kore a job and it does it through your own accounts. Gmail and Google Calendar, plus 22 services through their official MCP servers (Notion, Linear, Jira, Confluence, GitLab, Stripe, Supabase and more), each connected per user with OAuth.
  • Speech in 16 languages, and built for accessibility: screen readers, 200% text and reduced motion.

How we built it, and how we use Nebius and NVIDIA

Every "thinking" step runs on NVIDIA open models through Nebius Token Factory (EU endpoint):

Model Role
NVIDIA Nemotron 3 Super The main worker: interpreting (one-to-one and the group table, where one sentence becomes every language at the table in a single call), and Kore's tool calling over the app and the user's connected services
NVIDIA Nemotron 3 Nano The safety guard: checks every connector action before the user sees it, and says no when in doubt
Qwen3-Embedding A capability library in pgvector, so Kore only gets the tools that match what you said

We tested the smaller models too. Nano turned "half past eight" into 9:30, and one Nemotron model swung between half a second and ten seconds per step. Super got the times right and stayed steady, so it does the main work.

Around that:

  • Speech: Deepgram streaming speech recognition, with Aura and Cartesia voices.
  • Backend: FastAPI with WebSocket relays. One app protocol, a few routes: our own interpreter pipeline (speech to text, Nemotron, text to speech), a Voice Agent route for languages without a native voice, and the group table.
  • App: Expo and React Native, with a dark "aurora glass" look.
  • Data: Supabase Postgres with row-level security. Personal data and OAuth tokens are encrypted, and usage is metered per month. No audio and no transcripts are stored.

Measured end to end with real speech: about 1.1 seconds from the end of your sentence to the spoken translation, and 1.0 to 1.5 seconds for Kore to act and answer.

Challenges we ran into

  • A full voice agent is slow to pay for. A hosted voice agent bills for every connected minute, even in silence. We built our own pipeline for interpreting and for Kore, and cut the cost of a voice command by about 88% while staying near one second.
  • The model answered instead of translating. Early on it replied "please provide the Spanish text". Cleaning up the conversation history, an English system prompt and temperature 0 fixed it, and recorded test runs confirmed it.
  • Safety for real actions. Translation can never trigger an action. Every write action passes the guard and a confirmation card. Financial connectors only get tools the provider marks as read-only. Our own security review found a flaw in how OAuth accounts were linked, and we fixed it: the signed-in app now has to finish the link.

Accomplishments that we're proud of

  • A group table where 8 people with 8 languages can talk around one phone.
  • Real connectors with per-user OAuth instead of demos: 24 services reach a real login.
  • A business model that adds up: every paid plan stays profitable even if a user maxes out every limit, and a test fails the build if that ever stops being true.

What we learned

In voice AI, most of the cost and delay sits in the speech, not in the language model. Nemotron on Token Factory is fast and cheap enough to use more than one model per conversation, each for the job it does best.

What's next

Where we are heading, and what the pictures above show: everyone at the table with their own phone, linked over Bluetooth or QR, so each person hears their own language. Today SpeaKore does it with one phone in the middle. After that: calling agencies, with Kore joining the call as an interpreter; app texts in 7 languages and speech in 17; and SpeaKore VR, the hands-first version for Meta Quest.

About the source code

SpeaKore is a commercial product, so we are not publishing its source code, and we know this does not meet the hackathon's public-repository rule. We are submitting anyway, out of respect for the event and to share what we built and learned. The public repository holds the documentation, architecture and examples of how we call Token Factory. Judges are welcome to a test build, a login and private access to the code on request: [email protected].

Built With

Share this project:

Updates

Submission history