Waylo

Waylo is an AI that lives on your Mac and Andriod phone and guides you through anything - a pulsing red dot shows exactly where to click next, spoken aloud, step by step.

Comment

Inspiration

We kept watching the same thing happen. A parent stuck on "where's the button to attach a photo?" A grandparent giving up on an app because one screen looked unfamiliar. The knowledge to do the thing exists. What's missing is a patient guide sitting beside you, pointing at the screen. Our shipping product, Waylo, already does that with a talking red dot. But it planned the whole task up front and followed it blindly, so the moment the screen wasn't what it expected, or the goal was ambiguous, it fell over. A real guide doesn't do that. A real guide watches what's actually on screen, asks when it's unsure, and remembers what you told it last time. That's a Collaborative Partner, and that's what we built here.

What it does

Waylo Agent guides a person through any task on their phone or computer, one step at a time. It asks when it's unsure, and it remembers what you told it.

You give it a goal in plain words, like "make the text bold in Pages" or "turn on dark mode." Then:

  • It reads the current screen, plans the task with Gemini, and turns each step into a red dot on the exact thing to tap, while speaking the instruction.
  • After you act, it checks the screen again and keeps going: on to the next step, done when the goal is reached, or recover if the screen went off track, like a pop-up or the wrong app.
  • When your goal is genuinely ambiguous, it does the key thing a real guide does. It asks a short clarifying question with options ("Would you like to create a new document, or open an existing one?"), then adapts to your answer.
  • It never dead-ends. When a task finishes it asks what's next, and it remembers what you just did, so a follow-up like "now make it bigger" simply continues in context.

It runs on both macOS and Android over one shared agent brain.

How we built it

  • Genkit (Google's agent framework) is the core. services/agent.js defines the flows with structured output, which forces Gemini to return exactly our schema. planFlow returns a full step-by-step plan for a task, and nextStepFlow returns one decision at a time. There is no fragile "parse the model's prose as JSON" step.
  • Gemini 3.5 (gemini-3.5-flash, Gemini API) does the reasoning. It reads the live screen, writes the plan, chooses whether to ask a clarifying question, and grounds hard-to-find elements in the pixels.
  • Cloud Run hosts the agent as a stateless container (Node/Express plus Genkit) that scales to zero when idle and serves the live production URL.
  • Firestore is the persistent memory. Per user, it stores the answers to clarifying questions and the goals pursued, so a brand-new session inherits what the user already told the agent and never re-asks. Within a session, the follow-up loop also carries prior tasks into the next plan as context.
  • The macOS (Swift/AppKit) and Android (Kotlin) clients read the screen through each platform's accessibility APIs, draw the red dot, and resolve every step to exact pixels with a fast on-device cascade, calling the cloud agent for the plan.

What was built for this hackathon (disclosure)

Per the New Projects Only rule, the boundary is explicit. Built during the Submission Period (Aug 3 to 31, 2026), in the dedicated waylo_agent repo and the separate client repo: the entire agent backend (the Genkit and Gemini 3.5 planner, the clarifying-question flow, the per-turn agent), the orchestration (the follow-up session loop, clarify handling, session memory), Firestore persistence, the Cloud Run deployment, and the macOS client's agent wiring (Follow-up mode, clarify UI, Right-Command voice and typed feedback). Pre-existing components carried in (built before Aug 3, and named as such): the macOS and Android client shell, and the on-device detection pipeline (L0 Accessibility, L1 OCR, L2 and L2.5 YOLO) that turns a planned step into pixel coordinates. Standard frameworks used: Genkit, Google GenAI SDK, Express, SwiftUI/AppKit. The new agent layer lives in its own repo created within the window, so the commit history reflects the timeline.

Challenges we ran into

  • From "plan everything" to a partner that adapts. The original product wrote a full plan up front and followed it blindly. Rebuilding it so the agent grounds each guide in the current screen, verifies, recovers, and asks when unsure was the core shift, and it is what makes it hold up when reality diverges from the plan.
  • Making it a partner, not just a runner. Adding the clarify path meant teaching the agent when to ask, only when a wrong guess would derail the task, without turning into a chatbot that pesters an elderly user with questions.
  • Real persistent memory. Passing history per request is easy but ephemeral. Wiring Firestore so answers survive across sessions, and merging them back in so the agent doesn't re-ask, is what actually delivers "adapts based on past interactions."
  • A thinking model that truncated its own output. gemini-3.5-flash spends output tokens on internal reasoning, so a tight token limit cut the plan JSON off mid-string. Raising the output budget fixed it.
  • Cloud Run build quirks. A macOS-generated npm lockfile made the strict npm ci fail inside the Linux build image. Switching to npm install in the Dockerfile fixed it.

Accomplishments that we're proud of

  • A genuinely agentic experience: perceive, reason, act, verify, adapt, ask, and remember, not a scripted plan.
  • Memory proven live. A vague follow-up like "make it bigger" targets the whole screen when it has no context, but after the user formats text in Pages, the same words correctly keep it in Pages and target the selected text.
  • Gemini 3.5 and Genkit on Google Cloud, with Cloud Run and Firestore working together, satisfying every hackathon requirement and deployed to a public URL.
  • One agent brain serving both macOS and Android.

What we learned

  • Structured output (via Genkit) is a superpower. Forcing the model to a schema removed an entire class of "unexpected JSON" bugs and made the agent's decisions trivially machine-usable.
  • For non-technical users, an honest "which one did you mean?" builds more trust than a confident wrong tap. The clarify path is as important as the action path.
  • Memory is what turns an assistant into a partner. The difference between "helpful once" and "adapts to you" is entirely in what it remembers between sessions.

What's next for Waylo Agent

  • Deeper personalization from memory, using remembered preferences like font size, pace, and which apps they use, to shape how it guides, not just what it does.
  • Richer recovery, feeding screenshots and vision grounding straight into the agent loop for pixel-exact steps.
  • More surfaces, with the same Cloud Run brain behind Windows and iOS clients.
  • Multilingual voice, starting with Hindi, so the guide speaks the user's own language.

Built With

Share this project:

Updates