YourEva

Eva sees your screen, hears your voice, and operates your computer - so people who have difficulty typing and clicking can still do everything.

Comment

Inspiration

This project is deeply personal. Anmol has experienced firsthand what it's like to lose the ability to type — twice.

The first time, he developed a ganglion cyst in his wrist and had to undergo surgery. Recovery meant staying away from keyboards — no typing for about a month. The second time was a shoulder injury — a labrum tear that put his arm in a sling. For that kind of injury, you can't use that arm for one to two months before physical therapy even begins, and long typing sessions remain impossible for months after.

In both cases, the frustration was the same: he could think perfectly fine, he could speak perfectly fine, but he couldn't get his thoughts into his computer at the speed he needed. Tasks that took seconds — sending an email, searching something, filling out a form — became painfully slow or required asking someone for help.

And he's someone who eventually recovered. For the 17 million people with cerebral palsy or 14.5 million with spinal cord injuries, this isn't a temporary inconvenience — it's every day, for life. Even among everyday computer users, 1 in 4 experience RSI or carpal tunnel symptoms that force them to limit keyboard use. The tools available today — rigid voice commands, expensive hardware, or human assistants at $300-$5,200/month — don't solve this.

That's why we built Eva: an AI assistant that can see the screen, hear your voice, understand what you need, and do it through a natural conversation.

What it does

Eva is a voice-first AI assistant that operates your computer on your behalf. You speak naturally, and Eva:

  • Sees your screen — captures screenshots and analyzes them with Gemini Flash to understand what's on screen
  • Listens and responds in real-time — bidirectional audio streaming via Gemini Live API with sub-800ms response latency
  • Supports instant interruption (barge-in) — speak while Eva is talking, she stops immediately and listens. Critical for users with speech difficulties
  • Clicks, types, scrolls, and navigates — controls any app via pyautogui, keyboard shortcuts, and macOS accessibility APIs
  • Handles complex multi-step tasks — "Create a Google Doc, write what I dictate, share it with my professor, and email them" — Eva decomposes this into steps and executes each one
  • Learns from repetition — saves successful task patterns as reusable skills.
  • Verifies every action — after each step, a reflection agent re-checks the screen to confirm it worked before moving on
  • Has a warm, patient persona — never rushes, checks in during silence, responds to frustration with empathy

How we built it

Two-Layer Architecture

Building Eva required the core insight that voice and action need separate models:

  • The Voice Layer uses Gemini Live API for real-time bidirectional audio — it handles conversation, understands intent, and responds naturally
  • The Work Layer uses Gemini Flash for screen analysis and action planning — it sees the screen, decides what to click/type, executes, and verifies

The coordinator (voice) calls delegate_task() which bridges to a completely separate text model pipeline. Getting these two layers to communicate seamlessly while keeping the voice layer responsive was one of our biggest architectural challenges.

3-Tier Routing System

A lot of what makes humans efficient on computers is invisible to AI models — keyboard shortcuts like Cmd+Space for Spotlight or Cmd+L for the URL bar. We built a 3-tier router:

  • Tier 1 (Direct execution) — pattern-matched OS commands via keyboard shortcuts. No LLM call needed. Executes instantly.
  • Tier 2 (Skill replay) — if Eva has learned this task before, she replays the saved skill pattern.
  • Tier 3 (Full agent) — the complete WorkerV2 pipeline: LLM plans actions, captures screenshots, executes, reflects, and self-corrects.

For example, opening YouTube can be done entirely via shortcuts with zero vision calls:

Cmd+Space → type "Safari" → Enter → Cmd+L → youtube.com → Enter → / → type query → Enter

The Pill Overlay

A floating pill overlay sits at the bottom of the screen — always visible, never blocking. Users press their configured hotkey (Fn by default) to toggle the microphone. The pill animates with a waveform when listening and shows status during actions. We adapted the overlay and hotkey patterns from Voquill (AGPLv3) and substantially modified them for Eva's accessibility-focused use case.

Memory System

We built three types of memory inspired by Agent-S (Apache 2.0):

  1. Episodic memory — stores successful task trajectories
  2. Narrative memory — after 3+ similar episodes, distills a reusable strategy
  3. Procedural memory (Skills) — parameterized, named skills. "Search YouTube for {query}" stores one skill where only the query varies

Split Cloud/Local Architecture

Eva's agent brain runs on Google Cloud Run, built with Google ADK (Agent Development Kit) and powered by Gemini Live API for voice and Gemini Flash for the worker agent. Tool execution (clicking, typing, screen capture) happens locally on the user's Mac via eva/local. The cloud sends tool_exec commands over WebSocket, the local executor runs them via pyautogui/mss, and sends results back. This adds <200ms per tool call.

Google Cloud services used:

  • Cloud Run — hosts the agent brain (voice coordinator + worker)
  • Firestore — stores episodic memory, narrative memory, and learned skills
  • Secret Manager — securely stores Gemini API keys
  • Artifact Registry — Docker image versioning for Cloud Run deployments
  • GCS — DMG distribution for macOS app releases
  • Terraform — infrastructure as code for all GCP resources
  • GitHub Actions — CI/CD pipelines for automated deployment on every push

Building this required solving the sync/async bridge problem: ADK runs delegate_task() synchronously in a ThreadPoolExecutor, but WebSocket communication is async. We bridged them using threading.Event + asyncio.run_coroutine_threadsafe.

Challenges we ran into

1. The Barge-in Bug — Google's ADK had a bug (#4460) where the interrupted signal was silently dropped when Eva was streaming text alongside audio. We diagnosed it, wrote an auto-patch, and reported it upstream.

2. The Quick Mute/Unmute Problem — When users mute immediately after speaking, the audio buffer hadn't flushed to Gemini Live. Eva heard nothing. We solved this by sending a 1200ms silence buffer on mute, exceeding Gemini's 700ms VAD threshold to trigger end-of-speech detection.

3. Screen Coordinates Across Displays — Different screen sizes and pixel densities meant click coordinates varied. We normalize screenshots to 1280px width before sending to the model, then scale coordinates back. For the cloud deployment, we use macOS's native Accessibility API (AX API) for element grounding — it runs in ~50ms, is extremely precise, and doesn't require a vision model call. For local testing, we also experimented with the UI-TARS model for precision clicking as an alternative grounding approach.

4. Reflection and Self-Correction — Eva needs to verify each action worked. We built a reflection agent that classifies post-action state as DONE, ON_TRACK, or STUCK, with up to 12 retry steps.

5. macOS Permissions — Unsigned apps face brutal friction. We used native AVCaptureDevice (Objective-C via Rust FFI) for mic permissions, AXIsProcessTrusted() for accessibility, and built a guided onboarding flow.

6. Compiling the Sidecar — PyInstaller silently crashed due to missing hidden imports for uvicorn/fastapi/starlette. We had to explicitly collect all submodules and test in isolation.

7. Minimizing Latency — We optimized at every layer: Rust backend for audio/hotkeys, Tier 1 routing to bypass LLM, aggressive image compression, keepalive silence frames every 5 seconds, and pre-session settings drain during the 300ms before Gemini starts.

8. Echo loop — Eva was hearing herself speak through the mic and responding to her own voice in a loop. We solved this using WebRTC's built-in echo cancellation (getUserMedia with echoCancellation: true) so the mic only picks up the user, not Eva's audio output.

9. Cloud vs Local Latency Trade-off — In our split architecture, the agent brain on Cloud Run communicates with the local Mac through a remote tool executor — when the worker decides to click or capture a screenshot, it sends a tool_exec command over WebSocket to the local executor, which runs it and sends the result back. The cloud deployment works smoothly and is our production architecture. For users who want slightly faster response times during testing, the local setup reduces network round-trips. Both modes are fully functional.

Accomplishments that we're proud of

  • Found and patched an ADK bug (#4460) that broke barge-in for all Gemini Live users — not just Eva. Our auto-patch benefits the entire ADK community.
  • Sub-800ms voice response latency — Eva feels like talking to a person, not a chatbot.
  • The split architecture works — agent brain on Cloud Run, tools on local Mac, communicating via WebSocket with <200ms overhead. This is a reusable pattern for any desktop AI agent.
  • Skill learning actually works — Eva gets faster with repeated tasks.
  • Full DMG distribution — users download, install, and talk to Eva without ever opening a terminal. Onboarding handles mic, accessibility, and screen recording permissions with native macOS dialogs.
  • Terraform + GitHub Actions CI/CD — infrastructure as code for all GCP resources, automated deployment on every push to main.
  • A persona that respects users — Eva is warm but never condescending. She treats users as capable adults who happen to prefer voice input.

What we learned

  1. Gemini Live's native audio model is a paradigm shift — no STT/LLM/TTS pipeline means dramatically lower latency. Voice interaction finally feels natural.

  2. The gap between "AI can do it" and "a user can use it" is enormous — most of our time went into error handling, deployment, permissions, and UX polish. The AI works; making it accessible to someone who can't type is the real engineering challenge.

  3. Split architecture is the right pattern for desktop AI agents — agent brain in the cloud (instant updates, no user setup), tool execution local (mouse/keyboard access). The WebSocket bridge is simple and robust.

  4. Memory transforms an agent from a tool into an assistant — without skill learning, every task is a first-time experience. With it, Eva gets faster over time, just like a real assistant who learns your routines.

  5. Accessibility is a design constraint that makes everything better — building for users who cannot use a keyboard forced us to think harder about voice UX, error recovery, and patience. The result is an interface that's better for everyone.

What's next for YourEva

  • Closing the cloud latency gap — right now, running Eva locally gives the best experience because the split architecture adds network overhead on every screenshot and tool call. We're working on optimizing this with smarter caching, delta screenshots (only send what changed), and edge deployment so the cloud version feels just as snappy as local. The architecture is built for scale — now we need to make it fast at scale too.
  • Open-source local models (via LMStudio / Ollama) for zero-cost operation — no API key needed
  • Windows + Linux support — expanding beyond macOS
  • Custom voice training — Eva adapts to users with non-standard speech patterns
  • Wearable integration — voice activation via AirPods, hearing aids, or switch devices
  • Apple Developer Program enrollment — signed app for frictionless installation
  • Multi-user skill sharing — users share learned skills with the community (e.g., "how to order on Instacart" becomes a shared skill anyone can use)

Built With

  • agent-s
  • fastapi
  • github
  • google-adk
  • google-artifact-registry
  • google-cloud
  • google-cloud-run
  • google-firestore
  • google-gemini-live-api
  • google-genai-sdk
  • google-secret-manager
  • mss
  • mui
  • pyautogui
  • pyinstaller
  • pytesseract
  • python
  • react
  • rust
  • tauri-2
  • terraform
  • typescript
  • ui-tars
  • voquill
  • zustand
Share this project:

Updates

Submission history