Sintonizados

Similar a la busqueda de sintonizar una radio, sintonizado busca que el publico que le cuesta seguir las charlas en otro idioma, puedan sintonizar el contenido, las voces del orador

Comment

Inspiration

Sintonizados started with a simple accessibility problem: at a conference with many simultaneous talks, providing live transcription and translation quickly becomes expensive, operationally complex, and difficult to scale.

The name comes from the idea of tuning in to a radio signal. We wanted every person to be able to tune in to a talk regardless of whether the speaker is using English or Spanish.

While designing the system, we realized that accessibility is not just a word-for-word translation problem.

A human interpreter constantly decides what matters, preserves context and technical terminology, reacts to language changes, and only spends additional effort when something is ambiguous.

That became the main architectural idea behind Sintonizados:

use the minimum amount of intelligence necessary for each decision, while preserving the context of the conversation.

What it does

Sintonizados receives live conference audio and produces real-time subtitles that can be consumed from a browser or embedded directly into a video stream.

Each talk is modeled as an independent session.

The system currently supports:

  • multiple simultaneous conference sessions;
  • live audio capture directly from the browser;
  • progressive PCM audio ingestion;
  • OBS audio ingestion through RTMP;
  • real-time transcription using Gemini Live;
  • fragment-level language decisions;
  • English-to-Spanish translation using Google Cloud Translation;
  • preserving fragments that are already in Spanish;
  • live original and translated subtitles through Server-Sent Events;
  • a transparent OBS Browser Source overlay;
  • subtitle latency measurements per session;
  • reconnectable browser clients with a bounded subtitle history.

A deterministic demo mode also allows the complete transport, session and concurrency pipeline to be tested without consuming external AI services.

How we built it

The orchestration layer is written in Go.

Instead of implementing a rigid:

audio -> transcription -> translation -> subtitle

pipeline, Sintonizados is session-oriented and event-driven.

Conceptually:

Audio -> Gemini Live -> Transcript Events -> Session Knowledge -> Decision Engine -> Translation when necessary -> Live Subtitles

The system separates responsibilities through interfaces such as:

  • Transcriber
  • Translator
  • DecisionEngine
  • ReasoningEngine
  • SessionStore
  • EventBus

This keeps the domain independent from any particular AI provider.

Lightweight decisions with Laya

Laya Multilingual runs as an independent OCI service.

For each stable transcript fragment, the Go gateway can ask Laya to classify it as:

  • Spanish
  • English
  • mixed
  • unknown

Spanish fragments can pass through unchanged, while English fragments are sent to the translation service.

If Laya is unavailable or uncertain, Sintonizados falls back to a deterministic path and lets Google Translation perform language detection.

The integration is technically functional, although language-classification quality is still being evaluated and is deliberately not treated as solved.

Google AI services

Gemini Live handles the real-time speech transcription.

Google Cloud Translation handles the specialized translation task instead of using a large generative model for every subtitle.

This keeps the critical path relatively small and allows more expensive reasoning to remain optional.

API design

The public API is modeled first using Smithy 2.0.

Smithy is the canonical API contract and generates an OpenAPI 3.1 specification.

The Go implementation remains independent from Smithy and from generated API types.

Containers and local operation

The project is container-first rather than Compose-first.

Podman and OCI containers are used to run the application and its tooling without requiring Go, Java, Python or ML dependencies to be installed directly on the development machine.

Laya runs in its own container, while MediaMTX and FFmpeg are used as replaceable adapters for OBS/RTMP audio.

Cloud deployment

The production/demo architecture uses two Google Cloud Run services:

  • sintonizados: the public Go gateway;
  • sintonizados-laya: the private Laya decision service.

The gateway authenticates against the private Laya service using its Google Cloud runtime identity and an internal application token.

Runtime secrets remain in Google Secret Manager.

OCI images are stored in Google Artifact Registry and tagged with the exact Git commit SHA.

CI/CD

GitHub Actions performs the complete verification pipeline:

  • Smithy validation;
  • OpenAPI generation and validation;
  • Laya OCI build;
  • Go tests with the race detector;
  • formatting and vet;
  • concurrent session smoke tests;
  • OCI runtime contract checks.

After CI succeeds on main, the deployment workflow authenticates to Google Cloud using GitHub OIDC and Workload Identity Federation.

No permanent Google service-account JSON keys are stored in GitHub.

The exact commit is built, pushed to Artifact Registry and deployed to Cloud Run.

Challenges we ran into

Real-time audio lifecycle

Processing a live stream is very different from processing a completed audio file.

We had to handle:

  • partial and final transcripts;
  • ordered audio ingestion;
  • session termination;
  • Gemini EOF draining;
  • provider timeouts;
  • cancellation;
  • reconnecting browser clients;
  • simultaneous sessions.

One particularly subtle problem was distinguishing a real Gemini failure from the normal period where the application waits for the final transcript after the audio stream has ended.

Code switching

A conference talk does not necessarily have one language.

A speaker can alternate between Spanish and English even inside the same section of a talk.

That forced us to move language information from a session-level property to a fragment-level decision.

Mixed-language fragments remain one of the areas we want to improve.

AI decisions are not automatically reliable

Integrating Laya was technically straightforward compared with validating its behavior.

Our initial multilingual tests showed that a model returning a probability is not the same thing as having a reliable classifier.

Instead of hiding that limitation, Sintonizados keeps explicit fallbacks and treats model quality as something that must be measured.

Scaling stateful real-time sessions

Cloud Run can scale horizontally, but the current MVP intentionally keeps its SessionStore and EventBus in memory.

For that reason the gateway currently runs as a single instance.

Sticky sessions would not solve consistency by themselves, so horizontal scaling is intentionally postponed until session ownership, durable events and distributed state are implemented correctly.

Accomplishments that we're proud of

We built a complete vertical slice instead of only demonstrating isolated AI calls.

Live audio can travel through the system, become a transcript, be classified, translated when necessary, published to connected viewers and measured for latency.

The same project can also receive audio from a browser or from OBS and expose a transparent subtitle overlay for streaming.

We also kept the architecture reproducible:

  • local development uses OCI containers;
  • the API contract is versioned;
  • CI validates the complete project;
  • CD deploys immutable commit-based images;
  • cloud authentication does not depend on long-lived credentials.

Most importantly, the system can continue operating when one intelligent component is unavailable. AI models are adapters, not the architecture itself.

What we learned

The biggest lesson was that different AI problems deserve different tools.

Speech recognition, translation, routing and contextual reasoning are not the same task.

Using a large generative model everywhere would increase latency, cost and complexity.

Sintonizados therefore follows a simple principle:

specialized models first, generative reasoning only when necessary.

That changed how we think about live accessibility.

Instead of viewing the system as a translation pipeline, we increasingly see it as a real-time knowledge-transfer system.

The spoken language is only one representation of that knowledge.

What's next

The current Laya integration makes fragment-level decisions, but the larger architecture goes further.

Laya can evolve into a pre-router that chooses the appropriate processing path:

  • direct translation;
  • specialized terminology;
  • a domain-specific model;
  • additional context;
  • semantic reasoning.

A second decision stage could later evaluate whether a localized result should be published, retried or escalated.

Gemma is planned as an asynchronous reasoning and knowledge-enrichment layer, outside the critical subtitle path, for tasks such as:

  • technical terminology extraction;
  • entity recognition;
  • glossary construction;
  • contextual disambiguation.

Other planned improvements include:

  • better code-switching segmentation;
  • distributed SessionStore and EventBus;
  • durable session ownership;
  • additional languages;
  • HLS and additional streaming inputs;
  • SRT/VTT export;
  • automatic Gemini Live reconnection and renewal;
  • adaptive model routing.

The long-term goal is for Sintonizados to become more than a subtitle generator: a reusable open platform for transferring meaning between different accessibility representations in real time.

Built With

Share this project:

Updates

Submission history