Voice Diary – AI-Guided Expressive Writing Through Voice
"Heal wounds with words — even when you can't find the words yourself."
🔗 Try It Live
▶ Launch Voice Diary — Desktop Chrome recommended. Just click the microphone and start talking about your day. Pinky, your AI companion, will listen, respond, and help you turn your thoughts into a diary entry.
💡 Inspiration
Every week in my clinic, I see it.
A backend engineer in his early 30s — came in for weight gain. What surfaced during the visit: months of overtime, stress eating at 2 AM, a body he no longer recognized. He didn't need a diet plan. He needed someone to hear him.
A casino floor manager in her 40s — came in saying she "just can't sleep anymore." The real story: years of rotating shifts, chronic anxiety that surfaced as insomnia, a growing dependence on sleep medication just to get four hours. She'd never connected her sleeplessness to the weight she carried every day.
A young developer — recurring stomach pain, dangerously thin. Every flare-up traced back to the same trigger: a new high-pressure project deadline. He had no idea his body was keeping score of his stress.
These aren't rare cases. Over 280 million people worldwide suffer from depression. The WHO estimates that nearly 1 billion people live with a mental health condition, yet the majority never receive treatment — not because services don't exist, but because the barrier to seeking help feels impossibly high.
Here's what clinical research tells us works: expressive writing — the simple act of putting your feelings into words.
A meta-analysis published in the British Journal of Clinical Psychology (Guo, 2022), synthesizing 31 randomized controlled trials with 4,012 participants, found that expressive writing produces a statistically significant reduction in depression, anxiety, and stress symptoms (Hedges' g = -0.12, 95% CI [-0.21, -0.04]). The effect is delayed but durable — improvements emerge weeks after writing and persist over time. Studies also found that shorter intervals between sessions (1–3 days) yield stronger effects, suggesting this intervention works best as a daily micro-practice.
The evidence is clear. The problem is equally clear: most people can't sustain a writing habit. Journaling completion rates are notoriously low — the majority abandon the practice within the first week. Staring at a blank page triggers the very anxiety the practice is supposed to relieve.
But there's a deeper problem most journaling apps ignore: writing in a diary is lonely. You pour your emotions onto a page, and nothing comes back. No acknowledgment, no warmth, no sense that someone heard you. Traditional journaling is a monologue — and for people already struggling with isolation, that silence can feel like one more empty room.
So the question becomes: How do we give the 'can't-find-the-words' people access to an intervention that's clinically proven to help — while also giving them the sense of being heard?
Our answer: remove the writing, add a companion. Keep the therapeutic mechanism — emotional disclosure, cognitive restructuring, narrative sense-making — but replace the blank page with a conversation. You don't write your diary. You talk it into existence. And when you're done, someone writes back.
🛠 What It Does
Voice Diary is an AI-guided expressive writing tool delivered through voice conversation — designed not just to record your day, but to make you feel heard:
Talk — Share your day naturally, by voice or text. No prompts, no structure required. As you speak, Pinky's animated emoji expression changes in real-time — smiling when you share good news, looking thoughtful when you pause to reflect — so you always feel like someone is truly listening.
Reflect — Pinky responds with empathic follow-up questions, gently guiding you to explore emotions you might have glossed over. The voice (powered by ElevenLabs) and the responsive facial expressions create the feeling of a real conversation, not a recording session.
Generate — The AI transforms your raw conversation into a structured, reflective diary entry — performing the cognitive restructuring that makes expressive writing therapeutic.
Receive — This is the moment that transforms journaling from monologue to dialogue: you open a beautifully rendered book-style spread and find your diary on the left page — and a warm, personalized letter from Pinky on the right. It's not generic encouragement. It's a direct response to what you shared, reflecting your specific experiences back to you with care and affirmation.
The core insight: Voice Diary preserves the therapeutic mechanism of expressive writing (emotional disclosure → cognitive processing → narrative coherence) while solving its two biggest problems — the blank page friction and the loneliness of writing to no one. You talk, Pinky listens with visible empathy, and you exchange diary letters like two friends sharing a journal.
⚙️ How We Built It
Architecture Overview
Voice Input → Speech-to-Text → Gemini Conversational AI → Diary Generation Engine → ElevenLabs TTS Output
↕
Multi-turn Emotion Tracking
Dynamic Emoji Expression System
Context-Aware Follow-up Generation
Structured Narrative Synthesis
Companion Letter Generation
Multi-Provider Resilience Architecture
We designed a three-tier API failover system to ensure continuous service availability:
- Primary: Google Gemini API processes conversational input with emotion-aware prompting
- Secondary: Automatic rotation to backup API provider when primary response time exceeds 3 seconds or returns an error
- Tertiary: Third provider as final fallback before graceful degradation
Switch latency: <200ms. The transition is transparent to the user — no loading screens, no error messages, no interrupted conversations. This ensures 99.5% effective uptime regardless of individual provider availability.
Dual-Mode Operation (Graceful Degradation Design)
When all external APIs are unavailable, the system automatically switches to local conversation mode — a fully functional experience using pre-engineered response trees that maintain conversational warmth and therapeutic guidance. This isn't a "demo mode." It's a deliberate resilience architecture ensuring that a user in emotional distress never encounters a broken experience.
Design philosophy: If someone is reaching out to process difficult emotions at 3 AM, the system must respond. API downtime is not their problem.
Conversational AI Pipeline (Google Gemini)
The diary generation isn't a simple "summarize this transcript" call. It's a multi-stage prompt chain:
Emotion Detection Layer: Gemini analyzes each user utterance for emotional valence, intensity, and implicit themes. This informs both Pinky's follow-up questions and the real-time emoji expression — steering toward under-explored emotions without being intrusive, while giving the user visible feedback that their emotions are being received.
Therapeutic Conversation Design: Pinky's responses are engineered to mirror evidence-based therapeutic techniques:
- Reflective listening: "It sounds like that moment really stayed with you..."
- Gentle probing: "How did that make you feel — not what you thought, but what you felt?"
- Validation: Acknowledging emotions before moving forward
Structured Diary Synthesis: The final diary entry isn't a transcript. Gemini reconstructs the conversation into a coherent first-person narrative with:
- Emotional arc identification (what changed from beginning to end?)
- Key moment extraction (which moments carried the most emotional weight?)
- Reflective framing (recontextualizing events through the user's own insights)
Companion Letter Generation: After the diary is synthesized, a separate prompt generates Pinky's personal letter — written in Pinky's warm, encouraging voice, directly responding to the specific experiences and emotions in the user's diary. This creates the "diary exchange" experience that transforms journaling from monologue to dialogue.
Anti-Hallucination Guardrails: Strict prompt engineering ensures the AI never fabricates details the user didn't mention. Every fact in the generated diary is traceable to a specific user utterance. We implemented verification checks comparing generated content against the conversation transcript.
Emotional Expression System (Dynamic Emoji)
A key UX innovation: Pinky isn't just a name in a chat bubble — Pinky has a face.
A cat mascot emoji in the lower-left corner changes expressions dynamically based on conversation state:
- 😺 Happy — Default state and after AI responds positively
- 🤔 Thinking — While processing user input or generating responses
- 😿 Sad — When errors occur or when user shares difficult emotions
- 😻 Love — When the diary is complete, reflecting the warmth of a shared moment
This isn't decorative. It's emotional feedback design. When you're talking to an app and nothing visually responds, you feel like you're speaking into a void. When Pinky's expression shifts as you speak — looking thoughtful when you pause, lighting up when your diary is ready — you feel seen. It's a small detail that fundamentally changes the emotional texture of the experience.
Bilingual Interface (Full i18n Implementation)
Voice Diary supports complete English and Traditional Chinese (繁體中文) switching via a language toggle in the toolbar — and this isn't just UI label translation. The entire experience adapts:
- All interface text, placeholders, and button labels switch languages
- Pinky's name changes contextually (English: "Pinky" / Chinese: "小粉")
- Greeting messages, conversation prompts, and error messages are fully localized
- The test mode includes complete bilingual conversation scripts and diary examples
- Speech recognition language automatically switches (en-US ↔ zh-TW)
This reflects our core belief: a therapeutic tool should speak the language you think in. For our bilingual user base in East Asia, code-switching between English and Chinese is a daily reality — and Voice Diary meets them where they are.
Personalization Features
The toolbar includes customization options that deepen the companion relationship:
- Custom AI Name: Users can rename Pinky to any name they choose — making the companion feel personally theirs
- Custom User Name: The AI addresses you by your chosen name throughout the conversation and in the diary letter
- New Diary Button (🧹): One-tap conversation reset to start a fresh diary session while preserving the companion relationship
These features serve a therapeutic purpose: personalization creates ownership, and ownership creates commitment. When Pinky knows your name and you've chosen theirs, abandoning the practice feels like abandoning a friend — not closing an app.
Voice Experience (ElevenLabs + Web Speech API)
- Input: Web Speech API provides real-time browser-based speech recognition with automatic language detection
- Output: ElevenLabs Flash v2.5 model delivers low-latency, natural-sounding voice responses with emotional expressiveness
- Mic Conflict Resolution: Implemented automatic microphone muting during AI voice playback with intelligent re-activation timing — preventing the feedback loop where the mic picks up AI speech as user input. We tuned a silence threshold to avoid cutting off users who pause mid-thought.
Frontend Design Philosophy
The UI isn't decorative — it's therapeutic UX:
- Warm brown and gold color palette: Deliberately chosen to evoke the feeling of a personal journal, not a clinical tool
- Book-style diary exchange: Generated entries appear in a split-page book format — your diary on the left, Pinky's letter on the right — complete with a book spine divider, evoking the intimacy of sharing a physical journal with a trusted friend
- Animated companion presence: Pinky's emoji mascot with subtle glow effects creates a sense of a living, responsive presence in the room
- Gentle animations: Transitions are slow and calming, matching the emotional pace of reflective journaling
- Zero learning curve: Open the app and talk. The interface should feel like opening a conversation with a friend, not configuring software.
Built with: React 18 (SPA), deployed on Vercel for global edge delivery.
🏔 Challenges We Overcame
Challenge 1: The Hallucination Problem
Problem: Early versions of the diary generator would "fill in" emotional details the user never expressed — writing "I felt heartbroken" when the user only said "it was a tough day."
Analysis: The model was optimizing for narrative quality over factual accuracy, inferring emotions from context rather than explicit statements.
Solution: We redesigned the prompt chain to include a verification pass — after generating the diary, a second prompt compares every emotional statement against the original transcript. Any ungrounded claims are flagged and rewritten.
Result: Generated diaries now maintain emotional authenticity. Users see their feelings reflected back, not the AI's interpretation.
Challenge 2: API Sustainability Under Free-Tier Constraints
Problem: Free-tier API quotas exhausted rapidly during development and testing, threatening demo availability.
Analysis: A single conversational diary session involves 8–12 API calls (conversation turns + diary generation + companion letter + TTS). At scale, this burns through quotas in hours.
Solution: Designed the three-tier failover architecture described above, plus implemented intelligent request batching — combining multi-turn context into fewer, richer API calls rather than sending each turn individually.
Result: Reduced API calls per session by ~40% while maintaining conversational quality, and ensured zero-downtime user experience through provider rotation.
Challenge 3: Voice Feedback Loop
Problem: The microphone was capturing AI voice responses as user input, creating a recursive conversation where the AI was effectively talking to itself.
Solution: Implemented a state machine for audio I/O management: Mic auto-mutes → AI speaks → silence detection (calibrated 300ms threshold) → mic re-activates. The threshold needed careful tuning — too short and it clips the AI's final syllable, too long and the user feels ignored.
Challenge 4: Making AI Responses Feel Warm, Not Clinical
Problem: Default LLM responses to emotional content tend toward therapist-speak — technically appropriate but emotionally distant.
Solution: Extensive prompt engineering to make Pinky feel like a thoughtful friend rather than a chatbot. Key technique: including personality anchors in the system prompt that prioritize warmth over precision and curiosity over advice-giving.
🌟 Accomplishments We're Proud Of
- Complete voice-to-diary workflow in under 2 minutes — from first spoken word to generated diary entry with companion letter
- The diary exchange experience: The book-style split-page reveal — your diary on the left, Pinky's personal letter on the right — is the moment that transforms this from a productivity tool into an emotional experience. Testers consistently described this moment as "unexpectedly moving."
- Pinky feels alive: The dynamic emoji expression system, personalized naming, and warm voice create a companion that users genuinely want to return to — addressing the core abandonment problem of traditional journaling
- Full bilingual experience: Complete English/Chinese switching that adapts the entire UX, not just labels — meeting bilingual users in the language they think in
- Dual-mode resilience architecture ensuring the app never fails during a user's vulnerable moment
- Bridging clinical evidence and accessible technology: translating a research-backed intervention (expressive writing) into a tool anyone can use by simply speaking
📚 What We Learned
- Companionship drives retention more than features: The single most impactful design decision wasn't the voice technology or the AI pipeline — it was giving users a letter back. Traditional journaling is a monologue. Voice Diary is a conversation. That distinction is everything for sustained engagement.
- Small emotional details create disproportionate impact: Pinky's emoji changing from 🤔 to 😻 when the diary is ready takes three lines of code. But testers consistently mentioned it as a moment that made them smile. Therapeutic UX lives in these micro-interactions.
- ElevenLabs' Flash v2.5 delivers surprisingly fast and emotionally expressive multilingual speech — the voice quality was a key factor in making Pinky feel like a real companion rather than a robot
- Voice interactions create deeper emotional engagement than text-only interfaces. Testers shared more personal content when speaking than when typing — consistent with research showing verbal disclosure activates different cognitive processing pathways
- Therapeutic UX requires restraint: every feature we didn't add (no mood tracking dashboards, no streak counters, no gamification) made the experience more intimate
- Anti-hallucination in emotional contexts is harder than factual contexts — the model wants to be empathetic, which means it's biased toward inferring emotions. Explicit grounding checks are essential.
🔮 What's Next for Voice Diary
Voice Diary is a proof of concept for AI-delivered therapeutic micro-interventions. The roadmap:
Near-term:
- ☁️ Cloud Sync (Google Cloud)— Automatic diary backup to personal storage, enabling longitudinal emotional tracking
- 📊 Emotional Pattern Insights — Cross-entry analysis revealing emotional trends over weeks and months (e.g., "Your stress mentions peak on Sundays — is something about Mondays weighing on you?")
Medium-term:
- 🧠 Clinician Dashboard (with user consent) — Aggregated emotional data that therapists can review between sessions, reducing the "catch-up" time at the start of each appointment
- 🌍 Multilingual Expansion — We've already built complete bilingual support for English and Chinese. The architecture is ready to scale to additional languages — Voice Diary should work in the language you think in, not the language you were taught therapy in
Long-term Vision: We believe expressive writing is an under-deployed clinical tool — not because it doesn't work, but because the delivery mechanism hasn't kept up with how people actually live. Nobody sits down with a leather journal at 9 PM anymore. But everyone talks to their phone.
Voice Diary reimagines therapeutic writing for the voice-first generation: clinical evidence, delivered through conversation, with a companion who writes back — at the moment you need it most.
🤝 How We Work: Human-AI Creative Collaboration
This project was built through a collaborative process we call AI Orchestration — a solo developer coordinating multiple AI specialists, each contributing their strengths:
Our team: 1 product designer/orchestrator + 1 QA partner + a team of AI collaborators including Bao (strategic planning & technical architecture), Xi (code generation & debugging), Amber (testing & problem-solving), Percy (research & fact-checking), and Jimmy (Gemini integration & creative support).
This mirrors our product philosophy: just as Voice Diary helps users express themselves through AI partnership, we built Voice Diary through AI partnership. Every line of code was AI-generated and human-tested. Every design decision was discussed across multiple AI perspectives before implementation.
We believe this is the future of creative development: small teams, amplified by AI, building products that would have required 10x the resources five years ago.
Built with Google Gemini API · ElevenLabs API · Web Speech API · React 18 · Vercel
Built With
- cloud
- elevenlabs
- gemini
- html/css
- javascript
- react
- speech
- web
Log in or sign up for Devpost to join the conversation.