Memoir

An agentic interviewer you talk to, by live voice or text, that helps you write a life story, a book, or an article in your own words. It cuts and reorders what you said, and never writes for you.

Comment

Many people want to write something long (a life story, a book, an article) and never do. The blocker is usually not ideas; it is the writing process itself. Professional ghostwriters solve this by interviewing: they draw the material out in conversation and shape it afterwards. Most people don't have access to one.

The obvious alternative, asking an LLM to write it, produces fluent text that isn't yours. Ask for a childhood memory and you get "bustling" markets and "humid" evenings: plausible, generic, and untrue. For personal or expert writing, that defeats the purpose.

Memoir fills the interviewer's role. It is built on three principles:

  • Keep the writer's voice : the AI never writes for you.
  • Interview well : ask questions that draw the material out.
  • Edit without rewriting : point out gaps and suggest fixes, which you can accept or ignore.

What it does

Memoir is an interviewer and an editorial staff. You talk; it does everything except the writing.

  • Interviews you one question at a time. It goes deeper rather than moving on: the specific afternoon, the object, the thing someone said.
  • Remembers across sessions. People, places, events, timeframes. In session twenty it can ask about the radio you mentioned in session two. An article takes a session or two; a book takes months of them; either way everything persists per project.
  • Composes passages from your own words. It may cut disfluencies, reorder, and repair the grammar reordering breaks. It may not add.
  • Criticises its own drafts. A background critic re-reads chapters against everything known and reports contradictions, timeline conflicts, thin sections, unexplained references. It never edits; it points.
  • Plans the next session from coverage gaps and open findings, and tells you why it is asking what it is asking.
  • Shows its receipts. Click any sentence in the manuscript to see exactly what you said to produce it.

How we guarantee provenance

Every sentence the system produces can be traced back to something you actually said. It is allowed to cut your filler words, reorder what you said, and fix the grammar that reordering breaks. It is not allowed to add anything.

This is enforced in code; we do not rely on prompting the model for correctness. Before any sentence reaches you, every meaningful word in it is verified against your recorded speech. Small connecting words ("the", "and", "was") are allowed freely, since that is what makes rearranging possible. Names and numbers match exactly. If the model slips in a word you never said, the sentence is blocked, the model is told which word was invented and asked to try again, and if it still fails, the sentence is discarded rather than shown.

you said:   "So, um, we moved to Chicago when I was, I think, seven. And the
             thing I remember most is the smell of rain on hot concrete."

accepted:   "We moved to Chicago when I was seven. What I remember most is
             the smell of rain on hot concrete."

rejected:   "We moved to bustling Chicago, where the humid air carried the
             smell of rain."          ← "bustling" and "humid" were never said

In the app, you can click any sentence and see the exact words it came from.

Human in the loop: balancing autonomy and control

The interview itself is human-paced. The autonomy runs between sessions: when a session ends, four agents pick up the work unsupervised:

  • the Archivist files what you said into the memory bank and composes new passages,
  • the Reviser notices when new material belongs inside a passage you already have, and proposes a rewritten version as a diff for your approval,
  • the Critic re-reads the affected chapters against everything known and raises what is wrong or missing,
  • the Planner updates the coverage map and writes the next session's plan with a reason attached.

By the next session the manuscript has grown, contradictions are flagged, and a plan is waiting. None of it writes a word on the author's behalf.

How I built it

Architecture. The browser client talks to a FastAPI service on Cloud Run: REST for actions, server-sent events for the streaming interviewer and the live crew progress, and a WebSocket for duplex voice. Inside the service, an editorial engine holds the domain logic: the interviewer, the composer with its provenance gate, the four background agents, memory, planning, and the usage ledger. The engine is written against protocols in engine/ports.py; everything Google-specific (the ADK interviewer, the Gemini text, speech, live and transcription clients, the Firestore store) lives in one adapter directory, adapters/adk/, and CI enforces that nothing under engine/ imports a vendor SDK. That boundary is what made it cheap to add each new Gemini modality: live voice was one adapter, no engine changes.

Agent craft as plain markdown. The interviewing technique, the editing constitution, the critic's rubric and the life-coverage taxonomy live in skills/ as plain text. That is the most portable and most valuable part of the system: it can be read by a non-programmer, reviewed like prose, versioned in a diff, and loaded by any agent framework.

Google stack. Gemini 3.5 Flash-Lite via the Gemini API (the rules allow either path); the key is held in Secret Manager. Google ADK runs the live interviewer, which uses its session runtime and event stream. The background crew makes single structured calls and talks to Gemini directly, because wrapping a one-shot JSON call in an agent runtime buys nothing. The interview itself is multimodal: typed text, a speech cascade (Gemini transcription and TTS around the text interviewer), or native duplex audio with Gemini Live, and every mode leaves the same verbatim record. Cloud Run hosts the service; Firestore holds state, with every record scoped to an author and a project, so cross-author reads are structurally impossible. When a session ends, the editorial crew runs as a persisted, resumable background run, its progress streamed live to the page. Gemini 3.5 trancribe is used for STT.

Evaluation. A system whose central claim is faithfulness should be able to prove it. Two evals ship with the product: a faithfulness eval (lexical layer = the production gate re-run as regression; semantic layer = a judge catching distortions that survive a word-level check, like a dropped negation), and a critic eval scored against contradictions planted in synthetic authors whose biographies I wrote, so recall has ground truth.

Synthetic authors. A memoir only becomes interesting after many sessions, which is a problem for both testing and demos. So the demo data is generated, not faked: an LLM plays a person with a fixed biography and sits through full interviews with the real agents. Every sentence in the demo manuscript is genuinely traceable; click-to-source works because the data is real.

Challenges

Defining "your own words." A raw transcript is unreadable; rewritten text is no longer yours. The rule that held up: cutting, reordering, and grammar repair are allowed; adding content is not.

Enforcing it in code. Prompt instructions alone failed constantly. The working approach is a pre-display check: every meaningful word must appear in the source speech, with connecting words allowed and names/numbers matched exactly. Tuning which words count as "connecting" took iteration: too strict and grammar repair becomes impossible, too loose and invented adjectives get through.

Evaluating the critic. Recall and precision only mean something against ground truth, so we test it with synthetic authors whose biographies we wrote, containing planted contradictions with known answers.

Making live voice compatible with the guarantee. A duplex voice model holds the conversation internally, so the session requests input and output transcription and persists both sides verbatim, so a live conversation leaves the same record a typed one does.

What we learned

The interesting constraint in agent design is often what the agent is forbidden to do. Every capability question in this project was easy; the restriction questions were hard, and they are where the product came from. "The AI does everything except the valuable part" is a design pattern with much wider application than writing.

Enforcing a restriction requires owning the agent loop. The provenance gate has to sit between the model and the author, inspect every sentence before it is shown, and send failures back. A lower-level agent library like ADK, which hands you the runtime and lets you decide what happens between turns, makes that possible. Fully packaged autonomous harnesses that own their loop are the right tool for other jobs, but a product defined by what the agent may not do needs the loop in the product's hands.

What's next

  • Elicitation quality: does the interviewer actually surface material a person would not have written unprompted? That is the research question under this product and it needs a study, not a script.
  • Semantic faithfulness at runtime, not just in evaluation.
  • Speech recognition that serves the accents of the people this is most for. The browser's recogniser is worst at exactly the voices that most need recording.
  • The people this is most for: elders, oral history projects, and people with a serious illness who want their story recorded, in their own words, while they can still tell it.

Try it

Tested at several levels: unit suites, recorded real-model outputs replayed in CI, UI tests against a simulated API, API-contract checks against the live schema, and judged end-to-end journeys (make experience). A full background crew run costs about half a cent on gemini-3.5-flash-lite. Exports: EPUB, markdown, HTML (print-to-PDF), plain text.

Built With

Share this project:

Updates

Submission history