Heard!

homepage of Heard!
Heard! in action

Inspiration

At 10 pm on the first night, we had two ideas and no decision. We wrote a design doc for each, half-built a demo for each, and every time one of us pitched, the other two pulled out their phones to check whether it was done. It usually had.

We realized we kept going back to whichever project we had sunk the most hours into, and none of us could look things up and argue at the same time. We looked at the AI meeting tools first. All of them are competent secretaries: they transcribe, they summarize, they send you the action items. None of them will sit in the meeting and think with you.

We wanted a teammate who would sit in that meeting with us, take the notes, do the research while we talked, and tell us when we were about to rebuild someone else's project. So we built Heard!, the thing we wished had been in the room.

What it does

Heard! is an ambient agent teammate that sits in a working meeting. It writes down the ideas and the threads of discussion, looks things up, brings information to the table, answers questions, and joins the discussion when it has something to add.

Takes the meeting notes, live, as people talk.
Sorts what comes up — ideas, projects, points of discussion — into cards on a shared board.
Investigates when needed: researches an open question and brings back supporting information and options.
Speaks up when it should, joining the discussion with an opinion of its own.
Answers questions from the room, in real time.

How we built it

We started from the demo, 、wrote out the two-minute meeting we wanted Heard! to sit in, worked backward from what had to happen on screen and when. The first thing we measured was speed. We benchmarked Nemotron models on NVIDIA NIM with reasoning on and off, and the numbers set the design: a card has to be on the board within five seconds of someone naming an idea, so nothing slow can be on that path.

Next we designed the architecture from the measurements. A fast Nemotron classifier (Super 120B, reasoning off) reads every few seconds of speech and sorts it: new idea, question, someone talking to Heard. A second Nemotron agent runs in the background and rewrites the meeting notes every twenty seconds. A Claude front desk, running as one long-lived Claude Agent SDK session, makes every decision the room can hear: answer, dispatch a research sub-agent, or stay quiet. Speech in and out is ElevenLabs, Scribe v2 for transcription, and streamed TTS for the voice. Then we built the harness because the model always tries to say something, which is annoying during a real meeting.

Finally, we wrote most of the code with Claude Code and tested the whole pipeline. During development, humans wrote most of the docs, architecture designs, implementation specs, and locked down code behavior with tests, while Claude Code implemented the code.

Challenges we ran into

It's all about ideas. We keep pushing back ideas and coming up with new ones with our own biases. But we are proud that we finally turned this big challenge into the whole project.

Accomplishments that we're proud of

We are surprised that the NVIDIA Nemotron classifier works so well that during our testing, most of the unrelated chats and debugging chats are well filtered out, and the AI agent doesn't respond to those at all! But everything related to a real project discussion, decision-making, and new ideas is all correctly flagged and recorded.
We are proud of the response speed Heard! has. We can reach near real-time asking and answering questions, investigating, and getting a reasonable result in 30s,
Nothing to the codebase itself, but our previous AI harnessing experiences make the whole project extremely easy to implement and very robust.

What we learned

Learned to use Nemotron APIs and ElevenLabs from scratch.
Learned to use the Claude Code SDK.
Dive deeper into agent loops and architecture design for an agentic system.

What's next for Heard!

Maybe persist past memories and make a personal database for all the meeting histories.
Potentially rewrite and refine the project to publish as open-source software.
Build start-ups that actually solve people's needs.