ZUUMLY

ZUUMLY is an AI-powered screen assistant that lets you share your screen, capture snapshots, and get instant voice or chat guidance in real time.

Comment

Inspiration

  • Zoom showed us how powerful frictionless screen sharing can be for communication.
  • Cluely inspired the idea of always-available AI help during real tasks.
  • We wanted to combine both into one product: screen context + instant AI guidance.

What it does

ZUUMLY lets users share their full screen or a specific window, capture snapshots (including selected regions), and ask an AI for contextual help through chat or voice. Users can attach snapshots to questions, get streaming responses, and hear AI replies with text-to-speech. Accounts and chat history are saved so users can continue conversations anytime.

How we built it

  • IDE: VS Code
  • Front-end: Electron + Vite + React + TypeScript
  • Database: Firestore
  • Authentication: Firebase Auth
  • LLM, transcription, TTS: Groq SDK
  • API fallback: Netlify Functions for chat, transcription, and speech endpoints

Challenges we ran into

  • Balancing privacy and usefulness when handling live preview and snapshots.
  • Prompting the model so it only describes visuals when the user explicitly asks.
  • Keeping streaming AI responses smooth while persisting chat history in real time.
  • Mapping region-selection coordinates accurately between preview and source resolution.
  • Supporting two runtime paths: direct SDK in desktop mode and serverless API fallback in web mode.

Accomplishments that we're proud of

  • Built an end-to-end desktop AI assistant with screen context, voice, and persistent chat.
  • Implemented live preview plus snapshot attach/detach workflows in a clean UX.
  • Added account/session controls including login/signup, clear chatlogs, and account deletion.
  • Integrated a complete voice loop: transcribe -> reason -> speak.
  • Reorganized the codebase into clearer pages/services architecture and improved documentation.

What we learned

  • Electron IPC boundaries and preload design are critical for security and maintainability.
  • Small metadata and prompt changes can drastically affect multimodal model behavior.
  • Streaming UX requires careful state orchestration to stay responsive.
  • Clear project structure and docs speed up team collaboration under hackathon timelines.
  • Building AI features is as much product design (intent, trust, clarity) as model integration.

What's next for ZUUMLY

  • Improve intent detection for when to use live preview versus snapshots.
  • Add smarter visual understanding features like OCR and richer screenshot annotation.
  • Introduce collaboration capabilities with shared sessions and team chat context.
  • Optimize bundle size/performance and add automated test coverage.
  • Ship a polished production release with stronger analytics and reliability tooling.

Built With

Share this project:

Updates