NerdLingo

• 

• 

• 

• 

Inspiration

I built NerdLingo around a very simple idea: accessibility at a conference should not depend on the language you speak or the stage you are attending.

Nerdearla already makes a great effort to make its content accessible, but what happens when several talks are happening at the same time?

A standard transcription tool may work very well for one speaker or one room, but a real conference is much more dynamic. There are multiple stages, different audio sources, live audiences, streaming tools, production teams, and people who need to access that information from their phones or computers in real time.

That became the main idea behind NerdLingo: instead of building just a transcription demo, I created something that could feel like part of the real infrastructure of a conference.

NerdLingo is an open-source real-time transcription and translation platform designed for multi-stage events.

What it does

NerdLingo listens to the audio of a live session, transcribes it, translates it when needed, and distributes the captions in real time.

The audience does not need to install anything. They can open the platform from their phone or computer, choose the stage they want to follow, and switch between the original transcription and the translated version.

Meanwhile, the production team has a different interface where they can control each stage independently, select the audio source, configure the source and target languages, monitor the session, observe caption latency, detect errors, and follow what is happening throughout the event.

NerdLingo also provides transparent caption overlays that can be added directly to OBS or vMix, so the same captions shown to the audience can also become part of a live stream.

One of my favorite parts of the project is that NerdLingo can, in a way, subtitle its own demo: while I speak in Spanish, one stage can translate my narration into English and display it directly in OBS, while another stage simultaneously processes the audio from another talk.

How we built it

I built NerdLingo as a TypeScript project with a Next.js frontend and a Node.js real-time backend.

The browser captures live audio using the Web Audio API and AudioWorklet. The audio is then processed and sent to the backend through WebSockets, where each active stage gets its own independent Gemini Live Translate session.

That separation became one of the most important parts of the architecture.

Instead of creating one AI session for every audience member, NerdLingo processes each stage only once and distributes the resulting captions to everyone connected to that stage.

This makes the system much closer to something that could realistically scale for a conference.

The frontend is deployed on Vercel, the backend on Render, and completed transcripts can be stored privately in Google Cloud Storage.

The entire project is open source under the MIT License.

Challenges we ran into

The biggest challenge was not simply getting an AI model to transcribe and translate audio.

The real challenge was making multiple live audio sources work at the same time without interfering with each other.

To test this, I created two real audio pipelines running simultaneously:

• one stage received the audio of a real Nerdearla talk through a virtual audio device;
• another stage received my physical microphone at the same time.

Both sessions were transcribed and translated independently without mixing their captions.

Seeing both stages running at the same time was one of the moments when the project stopped feeling like a simple prototype and started feeling like a real conference system.

Another challenge was latency.

In repeated tests, the first captions usually appeared in around 4–5 seconds. Instead of hiding that behavior, I added first-caption latency directly to the production dashboard so operators can monitor it in real time.

I also had to think about the system from two very different perspectives: the audience needs something extremely simple, while the production team needs much more information and control.

Accomplishments that we're proud of

During the Vibeathon, I was able to build and validate:

• real-time English ↔ Spanish transcription and translation;
• multiple independent stages running simultaneously;
• isolated physical and virtual audio inputs;
• responsive captions for the audience;
• source and target language configuration;
• a production console;
• a global monitoring dashboard;
• first-caption latency metrics;
• transparent caption overlays for OBS/vMix;
• transcript export;
• private transcript storage in Google Cloud Storage;
• production deployment;
• and complete open-source documentation.

But the part I am most proud of is not one specific feature.

It is that NerdLingo became more than an AI transcription demo.

It became a small but complete accessibility infrastructure for live events.

What we learned

This project taught me that building a product with artificial intelligence often has much more to do with the system surrounding the model than with the model itself.

AI can generate the transcription and translation, but a useful real-time product also needs reliable audio capture, isolated sessions, real-time communication, monitoring, error handling, audience distribution, production controls, streaming integration, transcript persistence, and a clear deployment strategy.

I also learned how important it is to design for the people operating the system, not only for the final user.

For the audience, NerdLingo should feel simple: choose a stage, choose a language, and read.

For the production team, the system needs to provide much more visibility and control.

Designing those two experiences separately made the project much stronger.

What's next for NerdLingo

There are many directions in which NerdLingo could continue to grow.

Future versions could include more languages, automated stage provisioning, richer transcript formats, distributed shared state, better operational metrics, and integrations with conference management platforms.

My long-term idea is simple:

an open-source accessibility layer that any conference community can deploy, adapt, and improve.

Accessibility should not be an extra feature added at the end.

It should be part of the event from the beginning.

Built With

• ai
• api
• audio
• audioworklet
• cloud
• css
• gemini
• github
• next.js
• node.js
• obs
• react
• render
• storage
• studio
• tailwind
• typescript
• vercel
• vmix
• web
• websockets