Crewmate

Crewmate is a multimodal AI remote employee, not a chatbot, not a copilot. It listens to your voice, sees your screen, and acts like a co-worker working alongside you. No copy-pasting. Just talk.

Comment

Crewmate — A Live AI Coworker

Inspiration

AI models are genuinely incredible. But the interaction model is stuck. Every workflow looks the same: open ChatGPT, paste context, get a response, go do the thing, come back, explain context again. I kept acting as the middleman between the AI and the actual tools I use.

I wanted to build something that didn't need to be summoned — something that could sit alongside you during a real work session, listen, watch your screen, and when you ask it to do something, actually do it. Not draft a response. Not describe the steps. Do it.

What I Built

Crewmate is a voice-first AI operator platform. You start a live session powered by Gemini Live API — no typing, no tab-switching, just talking. When you ask it to handle a task, it hands off to a background orchestrator that routes work to 14 specialist agents (research, writing, communications, productivity) across 51 skills — while the voice conversation stays live and uninterrupted.

Real session example:

  • "Do a competitive analysis on our top three competitors and save findings to Notion." → Background agent web-researches, synthesizes, creates a Notion page.
  • "Screenshot this bug and raise it as a task." → Screenshot captured, ClickUp task created.
  • "Share that in #tech on Slack." → Done.
  • "Write a PRD for this feature." → Research agent gathers context, writing agent drafts in Google Docs.

All from voice. No copy-pasting.

How I Built It

The architecture has three layers:

  1. Live Layer — Gemini Live API streams bidirectional audio. The gateway listens for intent signals and dispatches background tasks without interrupting the conversation.
  2. Orchestrator Layer — A Node.js orchestrator routes tasks to specialist agents. Each agent is scoped (research, Slack, Notion, ClickUp, Google Workspace, browser automation). Browser tasks use Stagehand + Browserbase for AI-native web automation.
  3. Memory Layer — Every session is summarized and embedded with Gemini Embedding. Stored in SQLite with vector search. Next session, Crewmate already knows what you were working on.

Deployed on Google Cloud Run with a Firebase-authenticated backend.

Challenges

The hardest problem: the live session and background agents can't share the same thread. When Crewmate dispatches a task, it needs to stream task updates back into the live conversation without blocking audio. Building that async broadcast layer — and making it feel seamless — took significant iteration.

Memory was the other inflection point. Before persistent memory, it felt like a demo. After, it felt like a tool you'd actually use.

What I Learned

The bottleneck in AI productivity isn't model capability — it's the interface. The chat box, the copy-paste loop, the context re-explaining. The models are ready. The question is whether we're willing to rethink how we work with them.

Built With

  • calendar
  • clickup-api
  • express.js
  • firebase-authentication
  • gemini-2.0-flash
  • gemini-2.5-flash
  • gemini-embedding-2
  • gemini-live-api
  • gmail)
  • google-cloud-run
  • google-workspace-apis-(docs
  • node.js
  • notion-api
  • playwright
  • react-19
  • sheets
  • slack-api
  • sqlite-(better-sqlite3)
  • stagehand-(browserbase)
  • tailwind-css
  • vite
  • zustand
Share this project:

Updates

Submission history