Lumina

Inspiration

The inspiration for Lumina came from a simple question: What if our AI interactions felt less like a chatbot and more like a shared dream? Most AI applications are utility-driven or purely conversational. We wanted to create something more visceral and artistic—a "lucid dreaming companion" that weaves a rich tapestry of sight, sound, and story in real-time. We were inspired by the concept of "Dream Logic," where the surreal and the unexpected are celebrated as creative opportunities.

What it does

Lumina is a multimodal narrative engine that guides users through an interactive, surreal dreamscape:

• Multimodal Input: Users can interact via text, voice, or by uploading images of their surroundings, which Lumina instantly interprets as part of the dream's setting.
• Interleaved Output: For every segment of the journey, Lumina generates:
  • A brief, poetic narrative segment.
  • A vivid visualization of the scene using Imagen 3.
  • Lifelike voice narration using Google Cloud TTS.
• Infinite Choice: The story is non-linear; the user's choices and sensory inputs directly influence the emotional tone and direction of the dream.

How we built it

Lumina is built on a robust Google Cloud stack:

• Brain: Gemini 2.0 Flash (via Vertex AI) handles narrative generation and multimodal reasoning, processing text, image, and audio bytes in a single context.
• Vision: Vertex AI Imagen 3 generates high-fidelity illustrations of the dreamscapes.
• Voice: Google Cloud Text-to-Speech (Journey Voices) provides emotive narration.
• Frontend: Streamlit was used to create a responsive and immersive web interface.
• Infrastructure: The application is containerized with Docker and deployed on Google Cloud Run for serverless scalability.

Challenges we ran into

• Multimodal Orchestration: Synchronizing text, images, and audio while maintaining low latency was a challenge. We had to balance the high fidelity of Imagen 3 with the need for a responsive storytelling feel.
• JSON Reliability: Getting the LLM to consistently output perfectly structured JSON (containing the story, image prompt, and choices) required robust prompt engineering and custom parsing logic to handle markdown code blocks.
• Audio Processing: Integrating real-time voice input in a web environment (Streamlit) while ensuring byte-level compatibility with Gemini's multimodal window required careful handling of MIME types.

Accomplishments that we're proud of

• Sensory Synergy: We are incredibly proud of how the text, visuals, and audio come together to create a unified mood. It doesn't feel like three separate tools; it feels like one "voice" telling a story.
• Developer Accessibility: We built a "Lite Mode" using the Google GenAI SDK and offline TTS (pyttsx3), allowing developers without immediate GCP access to still experience the core story engine.

What we learned

• Hallucinations as a Feature: In most AI apps, "hallucinations" are a bug. In Lumina, they are a feature. We learned to lean into the unpredictable nature of LLMs to enhance the "dream-like" feel.
• The Power of Interleaving: Providing multimodal feedback simultaneously creates a significantly deeper emotional connection than simple text-based chat.

What's next for Lumina

• Streaming Dreams: Implementing the Gemini Live API for real-time, low-latency voice interruptions.
• Persistent Memory: Integrating Firestore to give dream characters persistent memories across multiple sessions.
• Shared Spaces: Enabling multi-user sessions where friends can influence a shared dream together.

Built With

• docker
• gemini-2.0-flash
• google-cloud-run
• google-cloud-text-to-speech
• imagen-3
• python
• streamlit
• vertex-ai