HoneyAI Live -- Real-time AI assistant

HoneyAI Live is a real-time AI assistant powered by Gemini Live that can see through your camera, listen to your voice, and guide you step-by-step to solve real-world problems.

Comment

Inspiration

The inspiration for HoneyAI Live came from the frustration of following static DIY videos or manuals while working with your hands. We’ve all been there: hands covered in grease or paint, trying to pause a YouTube video or flip a page. We wanted to build a "digital pair of eyes" that acts as a real human guide watching exactly what you’re doing and providing hands-free, real-time verbal instructions without you ever having to touch a screen. We also realized that this "pair of eyes" shouldn't just be for physical repairs, but also for digital ones helping users navigate complex UIs or debug code in real-time.

What it does

HoneyAI Live is a multimodal AI assistant that provides a hands-free, real-time "Smart Mentor" experience for both physical and digital tasks. Powered by Google Gemini Live, it:

Sees and Hears: Streams continuous video and audio to provide context-aware guidance. Live Screen Understanding: Acting as a digital mentor, HoneyAI can "see" your screen in real-time. This allows it to help with complex navigation, guide you through form filling, or even act as a Live Code Assistant. Instant Code Help: You can ask questions about your code on the fly without ever copying and pasting an error message or taking manual screenshots. HoneyAI sees the error as you do and provides instant verbal fixes. Dynamic Planning: Generates custom, hyper-specific step-by-step plans for any project, from plumbing to programming. Hands-Free Interaction: It uses "active listening" and Human-like Interruption. If you speak while the AI is talking, it will stop instantly to listen to your question and respond accordingly, just like a real human mentor. Visual Auto-Validation: Periodically analyzes your camera or screen feed to confirm if you've completed a step and automatically advances the plan. Long-Term Memory: Remembers past sessions and project history, allowing your digital mentor to resume your project seamlessly across different sessions.

How we built it

Core AI: We leveraged the Google Gemini Live API for real-time multimodal (Audio/Video/Screen) reasoning. We also used Gemini 3 for high-precision vision-based step validation and complex task planning. Frontend: Built with Next.js 15 and Tailwind CSS. We implemented custom Web Audio Worklets for high-fidelity audio streaming and a Screen Capture API bridge to stream display content alongside camera data. Backend: A FastAPI (Python) server orchestrates the media relays to Gemini Live, handling the vision analysis and state management for the Smart Mentor. Persistence & Cloud: Used SQLite with an async driver for memory. The stack is containerized and deployed on Google Cloud Run for low-latency global access.

Challenges we ran into

Multimodal Sync: Synchronizing real-time screen shares, camera feeds, and raw audio streams was a significant engineering feat. We optimized our binary WebSocket relay to handle high-bandwidth media without losing sync. Screen Perception: Getting the AI to understand small text in code editors or complex UI layouts in real-time requires a balance between frame rate and resolution. We tuned our vision-processor to prioritize clarity in digital tasks. Production "Bake-in": Configuring environmental variables for Next.js build-time resolution while ensuring secure cloud deployment was a key hurdle we solved.

Accomplishments that we're proud of

True Smart Mentor Flow: We achieved a seamless experience where the AI watches a coder work, identifies a syntax error, and speaks the solution before the coder even asks no copy-pasting required. Universal Hands-Free Guidance: Whether it's fixing a leaking sink or fixing a bug in Python, the tutor acts as a consistent, verbal guide. Low-Latency Interaction: The system reacts to visual changes or voice commands in near real-time, making the mentor feel truly "present."

What we learned

We mastered the art of real-time interleaved media streams across multiple inputs (Screen + Camera + Audio). We learned that "context is king" the AI is significantly more helpful when it can see exactly what the user is seeing without the friction of manual uploads. We mastered the nuances of Cloud Run deployment for high-performance, real-time media agents.

What's next for HoneyAI Live

Deeper IDE Integration: Building plugins for VS Code and IntelliJ to allow HoneyAI to directly interact with codebases it sees. Hardware Integration: Smart glasses support for an immersive "eyes-level" mentor experience. Collaborative Mentoring: Enabling multiple agents to work together on complex, cross-disciplinary projects.

Built With

  • aiosqlite-apis:-google-genai-sdk
  • gemini
  • google-cloud
  • google-genaiadk
  • googlecloudrun
  • liveapi
  • next.js
  • python
  • screen-capture-api-cloud:-google-cloud-run
  • secret-manager
  • sqlite
  • tailwind
  • tailwind-css-backend:-fastapi-(python)
  • typescript
  • web-audio-api
Share this project:

Updates

Submission history