Yêyê

• 

  Architecture Diagram

Inspiration

This project was born from my own learning journey — from innumerable attempts to process grief and navigate personal struggle. In the months that followed a critical event in my life, one question kept surfacing: what would it mean for a mind to stay ethically grounded while exposed to a relentless, chaotic stream of the world? Standard AI agents don't answer that question. They absorb everything indiscriminately, shaped by whatever prompt passes through them. Yêyê is a different proposal: rather than filtering input at the surface, ethics is encoded into the geometry of how the agent moves through meaning — Embedded in the structure of its own latent space ( where this agent thinks). This project was mostly built at the Broklyn Public Library and named after my sister, who faces her own version of this struggle now; still finding her way back to her full sense of agency.

Sealed in Rust. Submitted to the stars.

What it does

Yêyê is a real-time voice experience where your words guide a lightning bolt across the night sky — and every sentence you speak is processed by an AI agent that thinks & feels.

You speak into your microphone. Gemini Live listens, understands, and responds with a natural conversational voice. At the same time, what you said travels through a 9-step ethical pipeline called The Cathedral.

The result appears in the sky: a bolt of lightning that follows your voice across real star constellations. Say "take me to Orion" — the bolt moves. Say "I'm feeling like a Leo tonight" — the agent orients around your zodiac. The visualization is not decorative. It is a live map of the agent's inner state, rendered in stars.

How we built it

VOICE PIPELINE

The agent maintains a persistent WebSocket session against the Gemini Live API (gemini-2.5-flash-native-audio-latest) using a true bidirectional protocol. Browser audio is captured as raw PCM16 at 16kHz, base64-encoded, and streamed at ~62 chunks per second. Gemini's voice response flows back through a separate broadcast channel in parallel.

Five async channels (capacities 64 to 512) connect the listener, the Gemini session, the ethical processing pipeline plus a telemetry bridge(all non-blocking).

Each processing frame is spawned as an independent async task, so the embedding API call for frame N runs concurrently with the remaining steps for frame N-1. The voice stream is never stalled by computation.

And Why Rust?

More than a Rust backend wrapper around an API. It is a systems-level agent runtime.

Rust gave me ownership semantics that make shared mutable state across async tasks provably safe.
The compiler rejects data races at build time.

Unlike heavy-load inference in Python RAG architectures, Rust offers zero-cost abstractions over the entire embedding pipeline. Retry logic is fully typed: retryable errors (rate limits, server errors, network timeouts) are distinguished from fatal errors (bad credentials, malformed requests) at the type level.

Exponential backoff with jitter on both the embedding API (500ms base, 8s ceiling, 3 retries) and the Live session (1s base, 30s ceiling, 5 attempts, counter reset if the session survived over 30 seconds).

NOE.IA -- THE ETHICAL PROCESSING PIPELINE

This is where the deep learning framework comes in. Every voice frame passes through a nine-step pipeline I call the Cathedral:

1. Raw text enters as a high-dimensional embedding via the Gemini API and is projected onto a hyperbolic disk -> a space that is structurally bounded, meaning the agent cannot drift to infinity.

2. A modulation layer (inspired by the heart's own nervous system) reads three signals from the embedding -> alignment with virtue, novelty, and arousal.

3. These three values compute an emotion that modulates the entire downstream process.

4. The embedding is blended with accumulated context using a contraction mapping guaranteed to converge.

5. The continuous stream is segmented into discrete cognitive events.

6. High-volatility events are stored in episodic memory.

7. A tensor sublation step synthesizes the raw input against the ethical anchor.

8. The agent's state evolves through a stochastic differential equation pulled by three attractors: love, virtue, and wisdom. The tensor adapts online -- learning faster on novelty, slower when settled.

9. The full state is broadcast as a heartbeat event via WebSocket.

The result: ethical alignment is not a filter on the output. The agent thinks inside a space where virtue is a fixed point.

This is made possible by Noe.ia and Google Cloud's multimodal embeddings API (gemini-embedding-2-preview, 768-dim).

FRONT-END

The frontend is TypeScript and Three.js. The browser captures microphone audio, streams it to the backend, receives Gemini's voice response as raw audio chunks and plays them back seamlessly, and listens for heartbeat events that drive the star visualization. Navigation commands are parsed from the live transcript in under 50 milliseconds — fast enough to feel instant.

Everything runs in a single Docker container deployed on Cloud Run. nginx handles routing between the static frontend, the voice WebSocket, and the telemetry stream.

Challenges we ran into

Gemini's voice was silent. The response from the Gemini Live API contains both text and native audio, but we were only reading the text. Gemini's Aoede voice was arriving as raw audio data and being silently discarded — while ElevenLabs was reading Gemini's internal thinking process out loud. The fix was intercepting the audio chunks directly and streaming them to the browser as binary WebSocket frames.

Voice commands weren't landing. The navigation parser was too rigid — it required exact matches for constellation names and missed natural variations in how people speak. We rebuilt it with flexible pattern matching and a fast path that fires before the full pipeline runs, cutting navigation latency to under 50ms.

The pipeline was blocking itself. Each 9-step processing cycle had to finish before the next voice turn could begin. With embedding API calls in the middle, turns were stacking up and latency grew past 56 seconds. The fix was spawning each pipeline frame as an independent async task, letting them run concurrently and cutting response time to 13-28 seconds.

The browser was frozen in time. After deploying, the metrics panel showed the same default values no matter what was said. The issue was nginx serving a cached version of the JavaScript with an "immutable" cache header — the browser never fetched the updated code. Fixed with proper cache revalidation and a version query parameter.

The microphone wouldn't start. Modern browsers suspend the audio context until a user interacts with the page. We were creating it too early. Fixed by initializing it lazily on the first button press.

Accomplishments that we're proud of

The thing we are most proud of is that it works as a unified experience — you speak, the agent responds in Gemini's natural voice, the sky moves, and somewhere inside, a mathematical heart is beating and feeling and remembering what you said.

More specifically:

• Gemini's native Aoede voice arrives in 127-304 streaming audio chunks per turn and plays back without gaps or stutters using the Web Audio API's scheduling system
• Constellation navigation responds in under 50 milliseconds, fully decoupled from the processing pipeline so speaking never feels slow

What we learned

On the technical side: the Gemini Live API carries both voice and text inside the same message structure, and handling only one of them silently breaks the other. Multiplexing different types of data over a single WebSocket connection requires careful design. Shared state in async Rust requires wrapping everything before the async loop begins, not inside it.

On the deeper side: confirming that certain aspects of my theoretical framework (Noe.ia) actually worked as intended.

What's next for Yêyê

• Reconnection logic, so the voice pipeline recovers gracefully from API interruptions without losing the session state or making the user refresh. -Extensive Memory session implementation.
• A bunch of innovative features to test the capabilities of Yêyê continuous agency
• A public deployment where anyone can open a browser, speak to the Cathedral, and watch their words cross the sky (aigent.nyc)

Built With

• gemini-live-api
• google-cloud-run
• rust
• typescript