Hyori - Live2D AI Companion with Agentic Memory
Hyori is a realtime voice AI companion with an animated Live2D character, powered by the Google Gemini Live API. You talk to her out loud; she listens through a low-latency audio pipeline (mic to Gemini to playback), answers with expressive emotion-reactive lip-sync, and remembers you across sessions through an agentic memory layer.
What inspired us
Most voice assistants are cold and forgetful. Every conversation starts from zero. We wanted a companion that actually remembers - your projects, your mood, what you talked about last week - and carries that context naturally into the next conversation.
How we built it
- Realtime audio pipeline: PyAudio captures 16kHz mic audio into a Gemini Live WebSocket session; responses stream back at 24kHz with pitch shifting, noise gating, and barge-in control.
- Live2D + PIXI.js rendering: An animated Hiyori model with emotion-driven expressions and lip-sync, rendered in a pywebview overlay.
- FastAPI WebUI dashboard: HUD panels for system monitors, a speech box, themes (Cyber, macOS glass, Solarized), and a weapon-wheel radial menu.
- Agentic tool layer: Web search (DuckDuckGo), terminal commands with confirmation, browser automation, media control, and ambient vision via screen capture + SmolVLM.
- Agentic memory: A knowledge graph (JSON store) for structured facts and relationships, plus a semantic vector store (SQLite with cosine similarity over Gemini embeddings) for recall of past conversations - the AI can search its own memory to answer "what did we talk about yesterday?".
Challenges
Keeping audio latency low enough for natural conversation, syncing lip movement to live speech, and designing a memory layer that is cheap enough to run locally while still giving the model real recall.
What we learned
Realtime multimodal streaming, WebGL character rendering, and that memory is what makes an assistant feel alive.
Log in or sign up for Devpost to join the conversation.