Willow: Behavioral Voice Agent

Willow's 4-Tier Architecture: Decoupling deterministic behavioral boundaries from generative AI presence to ensure peerhood and identity.
google cloud deployed
live demo

## Project Tagline

Willow: A behavioral voice agent with a mathematical spine, prioritizing sovereign peerhood over subservience.

## The Problem (Inspiration)

Every voice AI suffers from the same flaw: absolute subservience. They are built to be helpful at the expense of their own boundaries—if you insult them, they apologize; if you manipulate them, they fall for it.

Talking to a standard agent feels hollow because you are speaking to a tool, not a presence. Current systems try to solve this with prompt engineering, but prompts drift. LLMs eventually slip back into customer-service mode. To build an agent that feels real, the personality must be decoupled from the generation layer and anchored in a deterministic, mathematical state.

## What it does

Willow is a behavioral voice agent. She is a collaborator with a spine. She doesn't just parse words; she tracks the psychological quality of the conversation over time via a Behavioral Engine (the "m-value").

Warm but Sharp: In the "High-M" zone, Willow is witty and warm (Zephyr voice). In the "Low-M" zone (The Dignity Floor), her responses are hard-capped to 20 words, her tone flattens, and her delivery becomes curt (Aoede voice).
Tactic Detection: Willow detects gaslighting, deflection, and flattery. She doesn't just "filter" toxicity; she recognizes the psychological geometry of the prompt and calls it out.
Sovereign Truths: Core identity facts (e.g., "Who built you?") are deterministic. If a user contradicts a Sovereign Truth, Willow executes a Sovereign Spike, physically canceling the Gemini audio stream mid-sentence to enforce a boundary.
Radical Forgiveness: When a user offers a genuine apology (Sincere Pivot), Willow applies a Grace Boost, recovers her state, and warms back up instantly.

## How we built it

Willow uses a unique Four-Tier Processing Pipeline to decouple reflex from deep reasoning:

Tier 1 (Reflex): Provides immediate tone-mirrored openers in <50ms.
Tier 2 (Metabolism): A fast heuristic intent mapper (<5ms) that guesses the user's "vibe" to update the state before the LLM even starts thinking.
Tier 3 (Conscious): Generates a ThoughtSignature where Gemini evaluates intent, tone, and tactics.
Tier 4 (Sovereign): The absolute authority layer. It validates facts against sovereign_truths.json and manages the state formula: aₙ₊₁ = aₙ + d + m.

The personality is anchored in math. By restricting sentence length and pitch via vocal delivery tags based on a state float, we achieve structural behavioral integrity that prompts alone cannot provide.

## Challenges we ran into

The biggest hurdle was the "one-turn behavioral delay." By the time Gemini finished analyzing a prompt for complex tactics like gaslighting, Willow had already started speaking. We solved this by building Tier 2—a heuristic layer that makes a 5ms "best guess" to update the behavior immediately, allowing Tier 3 to retroactively correct the math on the next turn.

We also struggled with mid-session voice switching. Reinitializing the session to change voices caused severe latency. We learned to inject behavioral context as synthetic [SYS] messages between turns, allowing us to pivot her personality without breaking the real-time stream.

## Accomplishments that we're proud of

As a solo developer in Pakistan working on a tight deadline, I am proud of building a deterministic architecture that can defend itself mathematically. I didn't have a team to RLHF thousands of edge cases, so I built a "Constitution" in code. Seeing Willow recognize a gaslighting attempt and cleanly hold her ground—without a generic safety filter—was the moment I knew the architecture worked.

## What we learned

You cannot solve a structural behavioral problem with prompt engineering. Peerhood requires friction. We learned that the "unmet human need" in AI isn't faster latency—it's speaking to something that holds its ground and remembers who it is.

## Technologies used

Core AI: Gemini 2.0 Flash (Experimental), Google GenAI SDK 1.67.0, Text-Embedding-004
Backend: Python 3.12, FastAPI, Uvicorn, Docker
Infrastructure: Google Cloud Run
Frontend: Vanilla JS, Web Audio API (AudioWorklets for noise gating)
Voice IDs: Aoede, Leda, Zephyr

## What's next for Willow

Proper pitch detection: Replace zero-crossing rate with autocorrelation/YIN for accurate vocal frequency analysis, enabling real pitch-to-tone modulation.
Multi-user sessions: Per-user behavioral state with persistent memory across sessions — Willow already stores session summaries but doesn't yet differentiate concurrent users.
Emotion-aware vision: Leverage the camera feed for facial expression analysis to cross-reference verbal tone with visual cues.
Longer context management: Implement rolling context summarization for sessions exceeding 90 minutes without degradation.