Bibliotalk

Inspiration

Voice makes learning feel human — but most voice assistants are fundamentally unauditable. For education, that’s a deal-breaker: a fluent voice that can’t show receipts is how misinformation becomes convincing.

Bibliotalk is our answer: a voice-first mentor experience where every non-trivial claim can be clicked back to the exact excerpt it came from.

What it does

Bibliotalk is a Matrix space with two room modes:

• Dialogue Rooms (private): live text + Gemini Live 1:1 voice calls with “Spirits” (agents). Every voice turn is mirrored back into the room as transcript + citations.
• Archive Rooms (public, read-only): ingestion-backed “living books.” For each source, we post a deterministic root summary and verbatim excerpt replies idempotently.

The wow: in Element Call, a Spirit speaks naturally, then we click a citation and show the exact ingested passage that matches the quoted substring.

How we built it

• Gemini Live is used as the real-time audio interface (ASR/TTS + live transcription streams).
• A platform-agnostic agent core does retrieval + citation selection + citation validation outside the model.
• EverMemOS stores and searches per-Spirit memory; we store canonical verbatim evidence locally for deterministic quote checks.
• A shared relational schema lives in packages/bt_store/ and is used by all services (SQLite dev, Postgres prod).
• Matrix UX comes from Element chat + Element Call; Matrix adapter + voice sidecar are Node/TS, core services are Python.

Challenges we ran into

• Trust vs latency: we needed live voice speed without sacrificing evidence checks.
• Streaming cancellation: interruptions must supersede generation instantly (no “talking over” the user).
• Determinism: idempotent archive publishing and “no-evidence” behavior must be stable across retries.
• Platform abstraction without overengineering: we designed contracts so Discord can be an adapter later, without re-implementing trust rules.

Accomplishments that we're proud of

• Cited voice: every spoken answer produces a posted transcript + structured citations in-room.
• Citations that mean something: quote-substring validation against stored verbatim segments, with cross-Spirit isolation.
• Two-mode safety: Archive Rooms never respond; Dialogue Rooms are interactive and interruptible.
• One schema, many services: ingestion + agents + adapters share bt_store to avoid drift and duplication.

What we learned

• “Voice UX” is an orchestration problem: cancellation, state, and observability matter as much as model quality.
• Trust features must be product features: “no evidence” is better than a confident lie.
• Platform adapters are easiest when contracts are explicit and the core stays platform-agnostic.

What's next for Bibliotalk

• Multi-participant dialogue rooms (room membership + roles; facilitator routing).
• Group calls + floor control (who speaks when; interruption policy).
• Archive publishing for more sources (books, podcasts, documents) with stronger idempotency guarantees.
• Discord adapter parity on the same core contracts and the same bt_store schema.

Built With

• gemini
• matrix
• python
• typescript