Tower of Babbage

Inspiration

Last year I participated in the vibeathon but put almost zero effort on it. This year, after already participating in a couple hackathons through the year, I was more motivated to put work into at least a decent submission. Plus, the theme seems very interesting, I teach classes for uni so I might use it myself to auto-generate subtitles for recordings at some point.

Despite my increased motivation compared to last year, I have a considerably decreased amount of time to actually work on it. What got me to actually participate was Wednesday workshops on some AI tooling, particulary CodeRabbit and Devin. I feel like I need to increase my vibe-skills so this is a nice opportunity to test some new workflows

What it does

Tower of Babbage turns the live audio of a conference stage into real-time captions in multiple languages, self-hosted and open source.

• Live transcription and translation in one step. The pipeline cuts the audio into chunks at natural pauses and each chunk goes to Gemini — or a local Gemma 4 via llama.cpp/vLLM — which returns the transcript in the original language and the translation in a single call. Verified for English <-> Spanish and Portuguese input
• Audience view built for phones. Pick a session, pick a language, read. Adjustable font size, live connection indicator, and color-coded speake labels when diarization tags are present.
• Many stages in parallel. One deployment runs several sessions at once; the pipeline schedules inference fairly across them and keeps per-session stats and latency accounting.
• Operator console. Audio comes from a browser microphone page with level meter and auto-reconnect, or from file replay for demos and testing.
• Production monitoring dashboard. Live status per session: latency, chunks processed, event log, and alarms when a session goes stale or errors.
• Technical glossary. Global and per-session term lists (kubectl, etcd, Nerdearla, speaker names) injected into the prompts, editable mid-session without restarting the run.
• OBS/vMix overlay. A transparent web page renders the subtitles to burn into the live stream as a browser source — no extra hardware.
• Transcript export and replay. Download SRT, VTT or TXT per language at the end of the talk, or watch the video back with captions in sync.
• Self-hostable. docker compose up brings up Postgres, the Go pipeline and the web app. Apache 2.0. Switching inference from Gemini to a fully local model is a config change, not a code change.

How we built it

I started with Claude Fable to produce initial documentation and design the app architechture. This is a workflow I tried a couple times before for other hackathons and it worked decently well. I passed it only the hackathon project description, with a semi-long prompt with my preferences for some details like stack and model. After a quick back-and-forth I left it to generate all the docs for the project, including a roadmap of the work to be done.

With that pushed to a repo I pivoted to CodeRabbit. First I told it to read the docs and generate issues based on the roadmap. I then went and started going through the issues one by one... until I ran out of usage. It was at enough to make all the scaffolding and project setup.

After that I switched to Devin (since I also got some free tokens this week). I squeezed every single token I could, I had 3-5 sessions running non-stop. It really helped that the work was already divided into issues with clear dependencies so I could parallelize as much as possible.

After a long night of work, by breakfast I had a good enough product and all that was left was finishing touches and recording a good demo

Challenges we ran into

The main one was that my first plan was to run it locally, but the only way to use the GPU in my desktop was to run it (and thus support) Windows. The initial setup was done in a unix environment and thus most things were fully incompatible with windows so a lot of additional work had to be done. I did plan to add support for cloud models at the end but I ended up having to fast-track that to have a portable, easy to run version for testing (and to free me from the cruel fate of using PowerShell).

This little kerfuffle with multi-OS setups made me quite thight on time, since I lost 2 hours in something avoidable. By 3am I was running multiple instances of different agent tools in parallel just to get things done a bit faster. The hardest part was coordinating tasks to make sure I could have as many running in simultaneous while not loosing too much time merging conflicts later

Accomplishments that we're proud of

The original plan ended up being so good. After that, all prompts were little more than "work on issue X" and all agents knew what to do so I was able to work in up to 4 taskk at the same time with no issues

What we learned

My main takeaway is that, if you're going to spend big in a session, the most worthwile way to do it is to make a comprehensive design and planning session, that way work is easier to divide up into independent tasks and edits aren't as complicated so cheaper models can perform relatively well.

While I feel like this is by far the less I've learned about actual code during a proyect, I did learn a lot about workflow and I feel this will let me do a lot better in projects where I'm not as time-constricted

What's next for Tower of Babbage

As I said before I might actually use it. Next week I will be teaching a class via meet so my aim is to test it there. There's some future steps in the repo documentation, I'll probably work on it on and off for a while for fun, I'm trying to put together a decent portfolio and I was lacking solo projects.

Built With

• claude
• coderabbit
• devin
• gemma
• golang
• t3
• typescript