Ambient

Ambient sits on your desktop, learns how you work, and handles the busywork before you even think about it. Scheduling, research, and analysis done instantly. Your employees, but 1000x.

Comment

Inspiration

I kept watching myself do the same thing every day from checking my calendar, drafting the same kind of email, alt-tabbing to look up something I already knew yesterday, pulling the same data for the same weekly assignments... None of it required any real thinking. It just felt like friction. And I realized every knowledge worker I know does the exact same thing. We're not slow because we're bad at our jobs. We're slow because 60% of our day is busywork that a sufficiently smart system could just handle.

I didn't want to build another chatbot you have to go talk to. I wanted to build the thing that already knows what you need before you ask. Like having the world's best executive assistant sitting behind your screen -- one that learns how you work, stays completely silent when you're in flow, and only shows up when it can genuinely save you time. Not to replace you, but to make you so fast at the menial stuff that you only ever do the work that actually matters.

What it does

Ambient is a desktop AI agent that sits on your Mac, passively watches your screen, and takes direct action the moment it detects an opportunity to save you time.

Three core things:

  1. Passive listening, smart surfacing. It captures your screen every 5 seconds and builds a real-time understanding of what you're doing -- which app, what task, who's involved, how urgent it is. During deep work, it stays completely invisible. No popups. No interruptions. It only appears when it detects a genuine high-leverage moment.

  2. Pattern learning. Ambient remembers your contacts, your scheduling habits, your common workflows. It builds a memory of how you work over time. The more you use it, the faster and more accurate it gets.

  3. Direct action via tools. When Ambient surfaces, it doesn't just suggest: it acts. It checks your Google Calendar and drafts a reply with a specific available time slot. It triggers a Hex analytics notebook pre-loaded with your workflow context. Every AI call routes through Lava for enterprise-grade cost tracking and usage metering.

The result: a scheduling email that normally takes 10 minutes of back-and-forth gets handled in 2 seconds. A data analysis request that would take 20 minutes of setup runs with one click.

How I built it

  • Desktop agent: Electron + React + Tailwind CSS. The app captures screenshots every 5 seconds using native macOS APIs and extracts the active window metadata.
  • Vision pipeline: OpenAI GPT-4o-mini processes each screenshot and extracts structured context -- the app name, the task, key entities (people, dates, amounts), urgency signals, and the work domain (engineering, email, finance, etc.).
  • Knowledge retrieval: OpenAI text-embedding-3-small generates embeddings for each screen context, stored in SQLite with vector search. This gives Ambient temporal awareness -- it knows not just what you're doing now, but what you were doing 2 minutes ago and how your workflow is evolving.
  • Synthesis engine: LLM pipeline (OpenAI primary, Anthropic secondary, Lava gateway tertiary) generates actionable recommendations as structured JSON -- including executable actions like create_calendar_event, draft_reply, run_hex_project, and copy_text.
  • User memory: SQLite-backed storage of learned preferences, contacts, and workflow patterns that persist across sessions.
  • Integrations: Google Calendar API for real-time availability checking and event creation, Hex API for triggering analytics notebooks, and Lava API as a transparent AI gateway for usage metering and cost visibility.
  • Monorepo: Turborepo + pnpm workspaces, with shared types across the engine, desktop app, and backend packages.

I built the entire thing solo over the hackathon.

Challenges I ran into

The hardest problem was knowing when to shut up. Every AI tool I've used interrupts constantly with notifications, suggestions, popups. It's annoying and it actually hurts productivity. Teaching Ambient when NOT to surface was harder than teaching it when to surface. I built a multi-signal gating system: LLM-scored relevance, domain-specific thresholds, temporal transition detection (did the user just switch contexts?), dwell-time analysis, and cooldown timers. The goal was to make Ambient earn the user's trust by being silent 95% of the time and being right the other 5%.

The second challenge was proactive over reactive. Early versions of the scheduling feature would surface a draft that said "When are you free?" - completely useless. The whole point is that Ambient should already know when you're free. Getting the AI to pull your actual calendar, find open slots, and write a reply with a specific proposed time required careful prompt engineering, calendar-aware context injection, and draft normalization to strip out any wishy-washy language.

Accomplishments that I'm proud of

The moment it actually worked end-to-end for the first time. I switched to Gmail, a scheduling thread was open, and 3 seconds later Ambient slid in with a ready-to-send reply that had a specific time from my actual calendar. I pasted it, hit send, and the meeting was booked in Gcal. The whole thing took 2 seconds for something that normally eats 10 minutes. That's when I knew this was a gamechanger.

I'm also surprisingly really proud of the invisible UX. It sounds counterintuitive to be proud of something the user doesn't see, but getting Ambient to stay silent during deep work and only appear at the right moment is genuinely novel. Most AI products optimize for engagement; Ambient optimizes for your time.

What I learned

That the hardest part of building an AI agent isn't the AI, it's the judgment. The LLM can extract context, draft emails, and call APIs just fine. The real engineering challenge is the decision layer: should I surface right now? Is this actually helpful or am I about to interrupt someone's flow state? That judgment is what separates a useful tool from an annoying one.

What's next for Ambient

  • Deeper integrations: Slack, Linear, Notion, Jira -- every tool a knowledge worker touches. Ambient should be able to create a Jira ticket, update a Notion doc, or reply in Slack without the user ever leaving their current task.
  • Team-level deployment: Right now Ambient learns one person's patterns. The next step is org-wide deployment where it understands team workflows, shared calendars, and cross-functional dependencies. A manager could see their entire team operating at 10x speed with full AI cost visibility through Lava.
  • On-device models: Moving the vision pipeline to local models for enterprises with strict data policies. Screen content never leaves the machine.
  • Voice: A natural language layer so you can just say "schedule that" or "run the analysis" without even clicking.

Built With

Share this project:

Updates

Submission history