-
-
Illustration of diary summary
Inspiration
The idea was inspired by my wife, who got tired of texting diary on her phone and wanted to have a new way to record daily memories, the agent is more than a
How we built it
We built this AI agent (NianNian) as a single-file PWA (no build step — all HTML, CSS, and JS lives in one index.html) backed by a Node.js/Express server on Google Cloud Run. The stack came together in layers: first a basic MediaRecorder → Gemini transcription loop, then a life-coach summarization prompt, then AI watercolor illustration generation, and finally — the hardest piece — the real-time voice companion NianNian, powered by the Gemini Live API over WebSocket. All user data stays in the browser (localStorage + IndexedDB); the server is purely stateless, acting as a secure relay between the client and Gemini's APIs.
Challenges we ran into
The Live API was brutally hard. Cold-starting a session took 5–20 seconds, so we pre-warm a session pool on the server to hide that latency. Getting bidirectional PCM audio right — capturing at 16kHz, scheduling playback at 24kHz with zero gaps — took careful Web Audio API work. The sneakiest bug: the model could hear itself speaking through the microphone, triggering false barge-ins. We solved it in software with an AudioWorklet RMS monitor that suppresses audio forwarding during playback — since we couldn't rely on hardware AEC in a browser context.
Cloud Run added its own wrinkles: the default 5-minute timeout silently killed long voice sessions, and a cold start mid-conversation drops the WebSocket entirely. Setting --timeout 3600 and --min-instances 1 was non-negotiable.
Accomplishments that we're proud of
NianNian genuinely feels like a companion, not a chatbot. She whispers back when you whisper. She asks follow-up questions without being prompted. She quietly saves meaningful moments to your diary via Function Calling — you don't have to ask. The emotional avatar mirrors the conversation in real time: she spins a thinking gear while processing, sends floating hearts when you say something warm, and throws confetti when you celebrate a win. We also got the whole thing to work offline-capable as an installable PWA with no framework, no bundler, and a single API key.
What we learned
The gap between "the API works" and "it feels natural" is enormous — and that gap lives entirely in timing, buffering, and edge cases. We learned that good voice UX requires solving echo cancellation, VAD sensitivity, gapless audio scheduling, and session warmup all at once, because failing any one of them breaks the illusion. We also learned that Gemini's multimodal capabilities let a small team punch well above their weight: transcription, summarization, image generation, real-time audio conversation, and function calling are each a product on their own — and we shipped all five in one app.
What's next for Niannian- Your Voice Diary Agent
The most natural next step is making Niannian truly multi-user — right now all data lives in a single browser's localStorage, which means you lose everything if you clear your cache or switch devices. Syncing entries to the cloud (with end-to-end encryption, since these are intimate personal thoughts) would unlock the app's full potential.
Beyond that, we want to push Niannian further as a longitudinal companion. Today she knows what you said this week — tomorrow she should remember that you've been stressed about the same project for three months, that your energy always dips on Sundays, that you light up when you talk about your cat. Grounding her in a personal long-term memory store would transform her from a clever chatbot into something closer to a real confidant.
A few other directions we're excited about:
Streaks and gentle nudges — a soft notification when you haven't recorded anything in two days, not as a guilt trip, but as NianNian checking in Mood timeline — visualizing emotional patterns extracted from entries over weeks and months, so you can see your own seasons Shared moments — optionally sharing a single illustrated entry card (not the full diary) with someone you trust, like sending a postcard The core thesis — that speaking is the most frictionless form of self-expression, and that AI should meet you there — still has a lot of room to run.
Log in or sign up for Devpost to join the conversation.