LiveBrief

• 

  Architecture

• 

• 

• 

• 

Inspiration

We wanted to build a different kind of AI research experience: one where getting an answer is only the beginning.

AI systems can produce convincing answers, but it can still be difficult to understand where those answers came from, inspect the supporting evidence, or know when the available information is insufficient.

LiveBrief was built around a simple idea:

Ask naturally. Investigate quickly. See the evidence.

We wanted the experience to feel conversational while keeping the underlying process grounded, traceable, and fast. That led us to combine a real-time voice and text interface with a native retrieval engine, evidence selection, grounded synthesis, citations, and explicit abstention.

What it does

LiveBrief is a real-time AI intelligence copilot for investigating a private knowledge base through natural conversation.

Users can:

• Ask questions using voice or text
• Retrieve relevant evidence from a knowledge base
• Generate grounded responses based on retrieved evidence
• Follow claims back to their supporting sources through interactive citations
• Inspect source material through the Evidence Ledger
• Continue investigations with contextual follow-up questions
• Receive an explicit abstention when sufficient evidence is unavailable

The goal is not simply to generate an answer, but to make the path from question → evidence → answer visible to the user.

Under the hood, Moss provides the native retrieval layer, combining hybrid BM25 and dense retrieval with MMR-based evidence selection before grounding and synthesis.

How we built it

LiveBrief uses a modular retrieval and grounding pipeline:

Voice / Text Query → Query Understanding → Moss Retrieval → Hybrid BM25 + Dense Retrieval → MMR Evidence Selection → Grounding & Synthesis → Citations & Evidence Ledger → Grounded Response

Moss is embedded directly into the LiveBrief application as the native retrieval engine rather than being treated as a separate AI agent.

The retrieval pipeline combines:

• BM25 for lexical matching
• Dense vector similarity for semantic retrieval
• MMR (Maximal Marginal Relevance) for selecting diverse and relevant evidence
• Deterministic grounding logic to constrain synthesis to retrieved evidence
• Citation-linked provenance so users can inspect supporting source material
• Abstention logic when the available evidence does not meet the required grounding threshold

The browser interface supports both text input and browser-native voice input. Conversational context allows follow-up questions to build naturally on previous turns.

We also added latency instrumentation and benchmarked individual pipeline stages rather than treating the entire application as a single latency number.

On the verified Moss benchmark, the native retrieval stage averaged approximately 0.4 ms over 50 iterations, with a p50 of approximately 0.26 ms and p95 of approximately 0.85 ms. MMR evidence selection was measured at approximately 1.5 ms.

These measurements describe the retrieval and evidence-selection stages, not the complete browser voice-to-response experience.

Challenges We Ran Into

One of the biggest challenges was balancing low latency with trustworthy retrieval and grounding.

Fast retrieval alone is not enough if the selected evidence is redundant, weak, or difficult to trace. We therefore had to design the retrieval pipeline around both performance and evidence quality.

Another challenge was making evidence visible without overwhelming the user. The Evidence Ledger and interactive citations became an important part of the interface because they allow users to move directly from a generated answer to the supporting source.

Voice interaction introduced another practical challenge. Browser-native speech recognition has its own endpointing and finalization delay, meaning perceived voice latency is different from backend retrieval latency. Profiling helped us separate these layers instead of incorrectly attributing all latency to the retrieval engine.

We also had to carefully handle unsupported questions. Rather than allowing the synthesis layer to fill gaps, LiveBrief uses an explicit grounding and abstention path when sufficient evidence is not available.

Accomplishments That We're Proud Of

We are proud of bringing several pieces together into a single investigation workflow:

• A conversational voice and text interface
• A native, low-latency retrieval engine
• Hybrid BM25 + dense retrieval
• MMR evidence selection
• Evidence-grounded synthesis
• Interactive citations and provenance
• Conversational follow-ups
• Explicit abstention
• Retrieval and latency instrumentation
• A browser-based investigation experience

Most importantly, we built LiveBrief around the idea that an AI-generated answer should remain connected to the evidence behind it.

We also validated the implementation with an automated test suite and browser smoke testing rather than relying only on a successful demo path.

What We Learned

We learned that building a useful real-time AI system is not just about choosing a model or making retrieval faster.

The quality of the overall experience depends on the entire path:

Input → Retrieval → Evidence Selection → Grounding → Response → Provenance → User Interaction

Working with Moss also pushed us to think carefully about what belongs on the hot path and how each stage should be measured independently.

We learned that latency should be decomposed rather than reduced to a single number. Native retrieval can be extremely fast while browser speech recognition, UI processing, and other layers contribute to perceived latency.

We also learned that grounding is a product feature, not just a backend implementation detail. Giving users a way to inspect the evidence changes how they can interact with and evaluate an AI-generated answer.

What's Next for LiveBrief

The next step is to expand LiveBrief from a focused knowledge-base copilot into a broader real-time intelligence workflow.

Potential directions include:

• Additional knowledge and data sources
• Multimodal evidence
• More advanced investigation workflows
• Richer evidence exploration
• Team and collaborative research features
• More deployment options for real-world use cases

The core principle will remain the same:

Fast answers are useful. Fast answers that you can investigate and verify are more useful.

Built With

• ai
• api
• bm25
• css
• fastapi
• git
• github
• html
• information
• javascript
• learning
• llm
• machine
• mmr
• moss
• next.js
• python
• rag
• retrieval
• search
• speech
• typescript
• vector
• web