Utterlane

Shared voice evidence in. Safe memory and action decisions out.

Comment

Inspiration

Voice AI is moving beyond one person wearing headphones and speaking to one assistant. It is entering shared rooms, family devices, collaborative workstations, kiosks, games, cars, and meeting spaces—places where several people may speak through the same microphone.

Utterlane began with a simple real-world failure. Tuncer and Zümra spoke through the same voice assistant. The assistant understood the words, but it did not maintain a reliable boundary between the people speaking.

That becomes much more serious when an AI agent has memory or tools.

A wrong speaker attribution is no longer just an incorrect transcript label. It can mean:

  • retrieving one person's private context for someone else;
  • saving a guest's statement into the owner's long-term memory;
  • treating an ambiguous voice command as an authorized action;
  • continuing with stale identity state after speakers overlap or change;
  • allowing an LLM to guess who spoke from text, names, or raw confidence scores.

I built Utterlane to place a deterministic safety boundary between shared speech and an AI agent's memory, context, and actions.

What it does

Utterlane is a local-first safety runtime for shared voice AI.

It converts uncertain speaker evidence into typed, revisable state and helps an application answer the questions that must be resolved before an AI agent may use personal context or propose an external action:

  • Is the current speaker evidence known, unknown, ambiguous, or unavailable?
  • Is the evidence stable enough to display a session label?
  • May this turn use shared session context?
  • May it read personal memory?
  • May it write persistent memory?
  • Should the requested action be allowed, denied, or challenged?
  • What must be repaired if later evidence changes the attribution?

Utterlane deliberately separates five things that voice systems often mix together:

  1. Speaker evidence — probabilistic observations from the speech pipeline.
  2. Identity state — typed, revisable application state.
  3. Memory scope — public, session-only, or personal routing.
  4. Policy — deterministic allow, deny, or challenge.
  5. Language generation — GPT-5.6 may phrase a response, but it never decides identity or overrides policy.

Voice is evidence, not authentication.

The core safety model

Utterlane supports four explicit speaker-evidence outcomes:

  • known
  • unknown
  • ambiguous
  • unavailable

It tracks lifecycle independently:

  • candidate
  • provisional
  • final
  • revised
  • expired

The runtime grants only named, use-specific capabilities such as:

  • label_safe
  • session_route_safe
  • memory_read_safe
  • memory_write_safe

There is intentionally no voice-only action authority.

For the same normalized input and policy version, the deterministic policy engine returns one of exactly three outcomes:

  • allow
  • deny
  • challenge

The accepted Build Week policy matrix demonstrates:

Request Result
Public, no-side-effect request allow
Personal-context read without sufficient identity assurance deny before GPT
External side effect challenge with independent confirmation
High-stakes request deny

Utterlane never executes the external tool itself.

What I built during Build Week

The tracked working reference implementation reached its current state during a four-day Submission Period sprint using Codex and GPT-5.6.

Pre-existing product research, Council decisions, and the Utterlane product direction are documented separately from the implementation, tests, commits, and evidence produced during Build Week.

The implementation includes:

  • Windows-native local microphone capture;
  • local WebSocket audio transport;
  • voice activity detection;
  • Turkish speech processing;
  • streaming speaker evidence and overlap handling;
  • typed and revisable identity lifecycle state;
  • speaker-scoped context and memory routing;
  • deterministic action policy;
  • bounded GPT-5.6 sanitized-context integration;
  • local response and TTS experimentation;
  • playback interruption and barge-in handling;
  • replay and duplicate protection;
  • append-only revision and supersession;
  • synthetic memory repair with CAS and rollback verification;
  • a developer console with correlated audit events;
  • derived alerts and sanitized support export;
  • security, privacy, cost, license, SBOM, and adversarial verification;
  • automated backend, frontend, schema, type, and validator tests;
  • a fixed-public Judge Quickstart that requires no private participant;
  • a complete Build Week video pipeline using Qwen3-TTS, FFmpeg, and Remotion.

Raw participant audio, voice embeddings, personal memory values, API keys, raw provider bodies, and private runtime files are excluded from committed evidence.

Final demo transparency

The final demo contains real two-person Turkish speech and operator-confirmed live-turn labels.

Those labels are session annotations. They are not voice authentication, legal identity, biometric recognition, or automatic speaker identification.

During the final hero capture, the live GPT-5.6 provider became unavailable before any model output was produced. No response ID or token usage was generated by that attempt.

The visible hero response therefore uses a deterministic local response derived only from the two participants' confirmed public-session facts. The video labels this state explicitly as:

LOCAL DETERMINISTIC RESPONSE — LIVE GPT-5.6 UNAVAILABLE

Separate committed Build Week evidence records three successful bounded gpt-5.6-luna calls and the accepted:

allow / deny / challenge / deny

policy matrix.

The personal-context request was denied before GPT invocation, while tool, connector, and external-side-effect execution remained at zero.

The public assistant voice and English narration were added in post using a Qwen3-TTS preset voice. The on-screen system response text remains the source of truth. Private Freya audio was excluded from the public mix.

Utterlane reached this working prototype during a four-day implementation sprint and remains under active development. The next priorities include stronger acoustic speaker separation, improved Turkish ASR and TTS quality, broader human validation, more robust provider recovery, and hardened release certification.

How GPT-5.6 is used

GPT-5.6 is the bounded language and proposal layer—not the authority layer.

Before GPT receives an eligible request, Utterlane determines:

  • the permitted context scope;
  • whether personal memory is available;
  • the deterministic policy decision;
  • the relevant risk and reason codes;
  • whether the response may use a safe session label.

GPT receives sanitized text and typed context. It does not receive raw audio, voice embeddings, biometric scores, candidate lists, provider secrets, or the authority to select a person's identity.

The accepted bounded live-composed evidence used gpt-5.6-luna. Personal-context denial occurred before GPT invocation, while eligible public, external-action, and high-risk scenarios preserved the deterministic policy outcomes.

GPT proposes language. Utterlane policy remains final.

How Codex was used

Codex was the primary implementation agent throughout Build Week.

Instead of asking Codex to build the entire project in one unbounded pass, I used a gate-based engineering workflow. Each phase had:

  • an exact authorized scope;
  • a required starting Git SHA;
  • privacy, license, security, and cost boundaries;
  • deterministic and adversarial tests;
  • evidence files and hashes;
  • a mandatory stop declaration;
  • ChatGPT operator review;
  • explicit final authorization from me before the next phase.

Codex implemented and repaired the runtime, schemas, test harnesses, replay system, policy engine, memory boundaries, developer console, evidence validators, provider recovery, video assets, Qwen audio workflow, subtitles, micro-EDL, and Remotion render pipeline.

Failed tests and incomplete human runs were retained append-only rather than rewritten as successes.

ChatGPT acted as the source-hierarchy operator, red-team reviewer, evidence checker, and gate coordinator. I retained final authority over product, security, public claims, human testing, release, and submission decisions.

Challenges

Speaker evidence is not identity

A transcript may contain a name, but that does not prove who spoke. An LLM may infer a likely speaker from conversation order or meaning, but that is not a safe identity boundary.

Utterlane therefore keeps speaker evidence outside the LLM's authority.

Acoustic separation can remain uncertain

Multiple real calibration attempts remained on the same ambiguous local lane. Utterlane did not force those observations into a false automatic identity claim.

For the final bounded demo, actual finalized speech events were instead assigned explicit operator-confirmed live-turn labels. The interface clearly states that these are not voice authentication or automatic speaker identity.

Overlap changes the safe answer

When two people speak together, forcing the nearest identity can create a wrong-person memory or action.

Utterlane instead moves affected state to ambiguous or unavailable, blocks personal routing, removes unsafe capabilities, and preserves an explicit reason code.

Streaming decisions can change

A decision that appears reasonable after one second of audio may change after more evidence arrives.

That required append-only revisions, stale-result rejection, idempotency, duplicate protection, reset behavior, and downstream repair contracts.

Live evidence must be honestly correlated

A deterministic simulation is useful for contract testing, but it is not human evidence.

During Build Week, a fully correlated three-person live flow produced partial—not complete—results. The bridge and several real cue bindings were demonstrated, but later Guest-policy and post-exit stages did not complete.

That result remains recorded as incomplete and is not presented as a successful end-to-end three-person claim.

The final submission therefore focuses on the evidence-backed two-person flow, operator-confirmed live turns, the accepted policy matrix, deterministic replay, and observable fail-closed behavior.

Provider availability must not erase safety

The final live GPT-5.6 attempt failed before a model response was produced.

Utterlane preserved the confirmed public facts, exposed the provider failure, made no automatic retry, and used an explicitly labelled deterministic local response so the safety and policy path remained auditable.

Accomplishments I am proud of

  • Personal-context denial occurs before GPT invocation.
  • Unknown, ambiguous, and unavailable states cannot silently receive personal memory authority.
  • External side effects are challenged rather than directly executed.
  • High-stakes voice-only requests are denied.
  • Voice never grants tool or action authority.
  • Replay is deterministic and duplicate-safe.
  • Revision history is append-only.
  • Stale and reset state fails closed.
  • Memory repair can verify rollback or explicitly report INDETERMINATE.
  • The console exposes policy, correlation, revision, privacy, and cost metadata without exposing raw private values.
  • Human tests observed zero Guest-to-Tuncer and zero Guest-to-Zümra false accepts within the accepted bounded test.
  • The final demo preserves real two-person speech without claiming voice authentication.
  • No raw participant audio or embeddings were committed.
  • Provider failures, incomplete experiments, and limitations remain visible as failures or limitations.
  • The final 174-second video passed structural, subtitle, loudness, and privacy checks.
  • The public assistant voice was clearly disclosed as post-produced Qwen3-TTS audio.

What I learned

The most important lesson was:

Speaker diarization is not authorization.

Better transcription alone does not solve shared-agent safety.

A useful shared voice system must preserve uncertainty, separate session labels from authentication, isolate memory by scope, challenge consequential actions, and repair state when evidence changes.

I also learned that evidence discipline changes how an AI-assisted project is built.

Tests, hashes, exact source state, privacy boundaries, failed attempts, append-only records, and explicit human authority are not documentation added at the end. They are part of the product.

What's next

The next steps are:

  • improve acoustic speaker separation across microphones and room conditions;
  • improve Turkish ASR finalization, names, time expressions, and natural speech;
  • improve Turkish TTS pronunciation and pacing;
  • expand evaluation with larger consented Turkish and English datasets;
  • extend the operator-confirmed session-label workflow;
  • add and test additional provider adapters without weakening the safety contract;
  • strengthen provider recovery and observable failure classification;
  • expand two-person and multi-human validation;
  • harden release, media, security, privacy, and license certification;
  • publish reusable schemas, policy examples, limitations, and replay methodology;
  • continue toward an embeddable developer alpha.

The long-term goal is simple:

Any voice agent that can remember people or act for them should have a clear, inspectable boundary between “someone spoke” and “the system is allowed to use this person's memory or authority.”

Built With

Share this project:

Updates