Cue

• 

  A sample conversation on Cue

Inspiration

When we think of agents, we often imagine entities we actively interact with—virtual assistants, chatbots, or autonomous collaborators. And while those are certainly compelling use cases, an agent can also play a more subtle role.

Sometimes, the most powerful agents are those that observe quietly, listening thoughtfully, and only stepping in when it truly matters.

Cue was born from this vision: a thoughtful, discerning AI that listens patiently and only speaks up when a significant factual discrepancy arises. We wanted to create a tool that doesn’t overwhelm but instead enhances trust and accuracy in the most subtle way possible.

What it does

Cue is a real-time audio fact-checking agent that passively listens to conversations and only intervenes when an egregious factual error is detected. Instead of constantly offering corrections or filling the air with noise, Cue provides gentle, well-timed prompts backed by credible sources. It acts as a silent guardian, offering truth only when it truly matters.

How we built it

We leveraged Deepgram APIs for real-time transcription, are able to recognize unique speakers, and receive the transcript in natural chunks.

We then send these statements to Mistral Small to evaluate statements on 4 parameters: Consequentiality, Factualness, Controversiality, Confidence.

If a threshold is exceeded on a mix of these parameters, an explanation is generated with ElevenLabs voices to and sent to the Cue for the user to interact with.

The entire pipeline operates asynchronously, ensuring low-latency responses without interrupting the natural flow of conversation.

Challenges we ran into

Building an agent that is more quiet than verbose was surprisingly difficult. Early versions of Cue were either too eager to fact-check trivial claims or too hesitant to intervene even when major inaccuracies surfaced.

Striking the right balance required fine-tuning our confidence thresholds and implementing a multi-step verification process to avoid unnecessary disruptions.

Maintaining low latency while performing live retrieval and verification posed a technical challenge that we addressed through parallel processing and caching strategies.

Accomplishments that we're proud of

We’re proud of building an AI that genuinely understands when not to speak—a rare trait in conversational agents, and it actually works! It handles real-world conversations with multi-speaker diarization through a clever use of web sockets.

During development, we accidentally left it on for like a whole hour, and it was fun to go through and see all of the actually incorrect things that we were saying.

What we learned

We learned that the hardest part of building agentic AI isn’t just about knowing the answer but about knowing when the answer matters. Developing Cue reinforced the idea that agency can mean thoughtful observation, not just active participation.

What's next for Cue

We envision Cue evolving into a trusted companion across different platforms, from video conferencing tools to podcast listening tools for public discourse monitoring. We plan to integrate memory and learning capabilities so that Cue can adapt to specific domains.

Ultimately, our goal is to create a silent ally for truth in any conversation.

Team

Justin Zhao (Eng) Frank Liu (Product) Matthew So (Eng) Jay Karp (Eng)

Built With

• deepgram
• elevenlabs
• fastapi
• mistral
• nextjs
• python
• react