Lumi — A Little More Joy in Every Day

Lumi brings joy and meaningful engagement to older adults through personalized games, guided exercises, Bible readings, family-shared photos and messages, and more—all through natural conversation.

Comment

Inspiration

For nearly nine years, we've built popular Alexa experiences at Voice Games. Along the way, we've seen how much older adults enjoy Alexa and how readily they've embraced voice for games, entertainment, and everyday interactions. We've also learned that building a voice experience is easy. Building a great one is hard.

At the same time, we've watched our own parents grow older and thought about how to help them stay engaged, active, and connected, especially when family lives far away.

These two experiences inspired Lumi.

Alexa already makes technology accessible through voice. We saw room for something more personal: a companion that doesn't just answer questions, but gets to know you, remembers what matters to you, and offers meaningful things to do together.

With Alexa+ and its agentic capabilities, we saw the potential to extend our experience in voice entertainment into that kind of companion—one that combines fun, gentle movement, faith, and family connection through natural conversation.

That's how Lumi was born: a little more joy in every day.

What it does

Lumi is a personalized companion that brings joy and meaningful engagement to older adults through natural conversation.

Lumi brings this vision to life through four core experiences:

Play — Games that engage and entertain

Lumi offers games personalized to individual interests. We've implemented Trivia and This or That, with more games on the roadmap.

Move — Staying active, one step at a time

Lumi guides users through gentle exercises with visual demonstrations, spoken instructions, and encouragement. Morning Stretch is implemented. Chair yoga, balance exercises, and other routines are next.

Reflect — Faith and meaningful reflection

Lumi offers Bible readings and interactive questions, allowing users to listen, follow along, and reflect through natural conversation.

Connect — Bringing families closer

Lumi Moments will let families share photos, videos, and messages through WhatsApp, iMessage, text, or email, making them available on Alexa+ enabled devices. We've built the interactive frontend, with end-to-end sharing as the next step.

A companion that gets to know you

Lumi remembers names and preferences, personalizes activities, and helps users move naturally between experiences.

Users can simply ask to play a game, stretch, or read the Bible together.

We're starting with a focused collection of experiences while building a foundation for many more activities.

One companion. Many possibilities. A little more joy in every day.

How we built it

Lumi is a voice-first, multimodal experience using AI agents and the Model Context Protocol (MCP), designed for future Alexa+ integration.

Our guiding principle: “The AI chooses what to do. Lumi decides what is true.” The agent interprets intent and calls MCP tools; Lumi's backend manages activities, rules, content, and state.

MCP architecture: A TypeScript server uses Streamable HTTP and JSON-RPC 2.0 with nine conversational and six app-only tools. Zod validates inputs; server-side logic drives activities.

AI and voice: Our browser-based Agent Simulator uses Vercel AI SDK and OpenAI o4-mini, with OpenAI speech-to-text, Silero VAD, and ElevenLabs speech synthesis. Langfuse and OpenTelemetry support tracing and evaluation.

Multimodal interface: React, Vite, and Three.js power games, exercise demonstrations, Bible readings, family Moments, and Lumi's animated character.

Data and infrastructure: PostgreSQL/Prisma stores persistent data, Redis handles sessions, Railway hosts backend services and databases, and Vercel hosts web apps.

Alexa+ integration: The Agent Simulator currently serves as MCP host. The same tools and backend are designed for future Alexa+ use, with account linking planned for persistent identity.

Challenges we ran into

Keeping voice and visuals synchronized: The AI agent sometimes repeated information or described activity steps that didn't exist. We redesigned MCP tool responses to include speech instructions, screen state, and context about what users had already heard, keeping conversations and visuals aligned.

Speech recognition and latency: Short spoken answers were difficult to recognize reliably. Deepgram with keyterm prompting didn't meet our needs, so we switched to OpenAI speech-to-text and added Silero VAD. We also moved from Gemini to OpenAI o4-mini after encountering latency and hallucination issues in our tests.

Improving agent reliability: Langfuse tracing, LLM-as-a-judge evaluations, and human testing helped us identify hallucinations and refine conversational behavior.

Alexa+ integration: Without third-party access to Alexa+ add-ons, we built a browser-based Agent Simulator to develop and demonstrate Lumi using the same MCP tools intended for future integration.

Accomplishments that we're proud of

We're proud to have brought games, guided movement, Bible readings, and family Moments together in one cohesive companion experience.

Our biggest accomplishment is the quality of the conversational experience we've developed. Lumi goes beyond responding to commands, handling natural responses, maintaining context, and guiding users through activities while keeping speech and visuals aligned.

Getting there required extensive iteration on conversation design, speech recognition, latency, and agent reliability. Combining AI-driven conversation with structured MCP tools helped us make interactions more natural and activity progression more predictable.

We're also proud of Lumi's animated character, accessible interface, and flexible architecture that allows us to introduce new experiences over time.

Most importantly, we've built something that feels like a companion rather than a collection of separate features.

What we learned

Building Lumi reinforced that great conversational experiences require more than a capable AI model. Speech accuracy, latency, context awareness, and synchronization between voice and visuals all matter.

We learned to balance natural conversation with predictable application behavior. The AI agent handles intent and dialogue, while structured MCP tools keep activities consistent without limiting conversational flexibility.

Langfuse tracing and evaluations helped us identify problems and improve interactions systematically.

We also learned that designing for older adults isn't about reducing capabilities. It's about making engaging experiences accessible through simple choices, clear visuals, and natural conversation.

What's next for Lumi

We're continuing to develop Lumi as an everyday companion, with more ways for older adults to stay engaged, active, and connected.

Bringing Lumi to Alexa+: When Alexa+ add-ons become available to third-party developers, we look forward to bringing Lumi to Alexa+ enabled devices.

Expanding activities: More personalized games, guided workouts, journaling, horoscopes, meditation, and daily reflections are on our roadmap, along with activities that make everyday life more enjoyable.

Lumi Moments: We'll build the end-to-end sharing experience, allowing trusted family members to send photos, videos, and personal messages through WhatsApp, iMessage, text, and email.

Growing with each person: We want Lumi to become more personalized over time, learning preferences and interests to make each interaction more meaningful.

Looking further ahead, we hope to help people capture life stories, discover new interests, and share moments across generations.

Lumi — A Little More Joy in Every Day.

Built With

Share this project:

Updates

Submission history