Lucevia

View Beyond What You See..!

Comment

Inspiration

Think about how Google Chrome offers an ambient "Ask Gemini" button right on the top corner of any webpage—ready to answer questions, explain concepts, and break down complex articles without forcing you to switch tabs or disrupt your flow.

Now look at television. Whether viewers are watching a university physics lecture, a rapid-fire political debate, a courtroom trial drama, a documentary, or tactical sports analysis on Netflix, YouTube, or Prime Video, they constantly encounter complex terms, dense formulas, or fast-moving context. Until now, the only option was pausing the TV, picking up a smartphone, and searching manually. We created Lucevia to bring that seamless ambient intelligence to the living room: a TV-native companion for Amazon Fire TV (Fire OS & Vega OS) that lets viewers pause at any timestamp and press "✦ Ask Lucevia AI" to immediately understand what they are watching.

What It Does

Lucevia runs as an ambient, non-intrusive TV overlay. Whenever a viewer pauses playback or triggers the agent:

  1. Multimodal Video Frame Vision: It captures the exact video frame at the paused timestamp and inspects the screen visually (identifying formulas, diagrams, on-screen text, speakers, and scene actions).
  2. YouTube & Streaming Deep Grounding: When streaming via YouTube, it fetches public oEmbed metadata and thumbnail previews to ground Gemini's answers in the exact subject matter rather than generic descriptions.
  3. Three TV Comprehension Modes:
    • Explain Simply: Delivers a structured domain breakdown and a memorable real-world analogy.
    • Show Visually: Generates a 4-phase conceptual flow and mental model diagram.
    • Check Understanding: An interactive multiple-choice question testing the viewer's grasp with instant explanations.
    • Ask Anything: A custom query bar where viewers can ask any specific question about that scene.
  4. Instant Playback Resume: One click or remote press smoothly dismisses the overlay and resumes the video.

How We Built It

  • TV-First Design (60:30:10 Rule): Engineered for 10-foot leanback television viewing using Obsidian Black (#040711), frosted slate glass panels with 40px backdrop blur, warm champagne gold focus rings (#f5c96a), and strict D-pad keyboard navigation.
  • Multimodal Backend Engine: Built with a lightweight Node.js backend integrating Google's @google/genai SDK (gemini-3.1-flash-lite) to process video frame captures and streaming metadata with sub-3-second response times.
  • Fire OS & Vega OS Architecture:
    • android-fireos/: Android system overlay service (TYPE_APPLICATION_OVERLAY) and accessibility service to intercept remote control shortcuts (KEYCODE_BUTTON_AI, long-press Select).
    • vega-os/: IPC event bridge and application manifest for Vega OS (Linux/React Native) television runtimes.

Important Prototype Note & GitHub Security Notice

Prototype Status: This project is currently a functional prototype developed during the hackathon. While the end-to-end multimodal loop, video frame analysis, YouTube grounding, and TV UI work end-to-end, it still requires further debugging, UI polishing across various TV screen aspect ratios, and domain-specific prompt tuning before commercial OS deployment.

API Key & Security: Due to security, privacy, and secret-scanning best practices, our private production Google Gemini API key has been excluded from the public GitHub repository. A .env.example template is provided so evaluators can run the project locally by inserting their own Gemini API key. (A standalone all-inclusive package with pre-configured dependencies is also available for direct evaluation).

Challenges We Faced

  • Television Screen Real Estate: Designing an agent UI that displays rich information (analogies, diagrams, quizzes) without clipping text, overflowing viewports, or obstructing the video context.
  • Multimodal Video Payloads: Optimizing high-resolution video frame captures on an offscreen HTML5 canvas to maintain high visual clarity while keeping payload sizes under 50KB for rapid Gemini inference.
  • Zero-Canned Intelligence: Ensuring Gemini dynamically grounds its responses in the actual video subject matter rather than discussing the internal mechanics of the agent application itself.

What We Learned

We learned that television AI needs to be significantly more ambient and friction-free than mobile or desktop AI. Viewers do not want a chat interface; they want rapid, glanceable, high-contrast mental models that answer their curiosity and get out of the way so they can continue enjoying their content.

Note

Please note that Lucevia is currently a proof-of-concept prototype that still requires additional engineering time for debugging across TV hardware aspect ratios, feature polishing, and specialized domain training—with a clear architectural roadmap designed to transition smoothly to Amazon's own LLMs and multimodal foundation models (such as Amazon Bedrock, Amazon Nova, or Alexa LLM) for native Fire OS and Vega OS deployment. Despite being in the prototype stage, Lucevia represents the single most compelling and untapped use-case for living room AI today: while modern AI has revolutionized laptops and smartphones, smart televisions have remained purely passive displays where AI is only used for search before playback begins. By bringing an ambient, non-intrusive comprehension companion directly to the screen via a single remote button press, Amazon Fire TV can transform passive watching into active understanding across lectures, sports tactics, news debates, and documentaries, unlocking a transformative living room experience that gives Amazon a decisive competitive edge in the smart TV industry.

Built With

Share this project:

Updates

Submission history