Cofounder

An agent for your Mac that notices the workflows you repeat, remembers them precisely, and takes them over.

Comment

About the project

Inspiration

Every automation tool starts with the same question: "What do you want to automate?" Almost nobody can answer it. The work we repeat is invisible from the inside. A freelance bookkeeper opens an invoice in Gmail, copies it into a spreadsheet and books it in her accounting tool, three times a morning, every morning. She would never call that "a workflow", and describing it precisely enough for a bot would take longer than just doing it. So the most repetitive work stays manual. It could be automated; it stays manual because nobody ever writes the spec.

We flipped the question around. Cofounder is an agent that watches how you work, notices what you repeat, and offers to take it over.

What it does

Cofounder is a macOS desktop agent with four layers.

  1. It observes behaviour, not content. Every two seconds, Cofounder notes which app is in front, the window title and the browser tab's URL, and writes a line only when one of them changes. It never records keystrokes, the clipboard or screenshots of your work. Observation is always visible and can be paused or erased in one click. E-mail addresses and long numbers in window titles are masked before any model sees them.
  2. It detects repeated workflows. Every 30 minutes, or when you click Analyse now, the last seven days of activity go to the Workflow Analyst, an AI agent. First, a deterministic miner extracts the app-and-site sequences that repeat. Then the agent does what a model is good at: it reads the evidence (timestamps, window titles), rejects navigation noise, checks what it already remembers, and saves only genuine workflows.
  3. It remembers precisely. Each workflow becomes a memory card with:
    • a human name, the ordered steps, the trigger and how often it happened;
    • an honest estimate of the minutes it costs per week, a confidence score, and the evidence it was built from;
    • an automation mission written for a computer-use agent.

Durable facts about how you work ("books supplier invoices in Pennylane") are reconfirmed over time, never duplicated. Everything is listed in the app and can be dismissed.

  1. It speaks up and takes over. A macOS notification tells you when a new workflow is found. Take it over hands its mission to Cofounder's computer-use executor, which:
    • plans the steps with the model;
    • reads the screen through the macOS accessibility tree, with a screenshot as a vision fallback;
    • drives the real mouse and keyboard, while an overlay narrates every step and a Stop button stays available.

Cofounder pauses its own observation while it works, so its clicks never count as your workflow.

How we built it

  • An evidence-bound agent. The Workflow Analyst is built with the open-source Strands Agents SDK (Python): one agent, one system prompt and six typed tools, get_activity_overview, find_repeated_workflows, get_workflow_evidence, recall_memory, save_workflow and remember_fact. The key design choice is that save_workflow must cite real candidate ids returned by the miner. The model can judge, name and explain workflows, but it cannot invent one the evidence does not show.
  • Deterministic mining (no model):
    • normalises and redacts events, then splits them into work sessions at 15-minute gaps;
    • collapses them into "stations" (an app, narrowed to a site for browsers) and removes alt-tab flicker;
    • mines non-overlapping sequences of distinct stations, and drops sub-sequences explained by longer runs as well as rotations of the same loop;
    • scores what is left by repetitions × length × distinct days.
  • Model-agnostic by design. In production the agent runs in the cloud behind an HTTP /invocations endpoint, with Claude as its primary model. The desktop app sends the activity in the request payload, so the cloud agent never needs access to the user's disk. The same agent, tools and prompt can also run fully locally on an open model through Ollama.
  • Resilience. A one-shot model probe with no retries detects a refused or throttled model in about 2 seconds. The agent then switches to the local model instead of stalling. If the cloud agent can't be reached, the app runs the same agent locally. The app always reports which model provider answered.
  • Desktop app. Electron, React, Vite and Tailwind, with:
    • a System Events observer (AppleScript);
    • a JSON memory store with atomic writes;
    • a JXA accessibility-tree reader with strict time budgets;
    • nut.js for mouse and keyboard, and a transparent overlay window.

Challenges we ran into

  • Workflow vs. navigation. Our first miner output was dominated by alt-tab laps (A→B→A→B) and by rotations of the same loop (B→C→A is the same work as A→B→C): a dozen "candidates" for one real workflow. Requiring distinct stations and removing rotations and sub-sequences brought it down to a short list the model can actually judge.
  • Keeping the model honest. We did not want an agent that describes plausible-sounding workflows. So persistence depends on evidence ids returned by the miner, and a recall_memory tool lets the agent update what it knows instead of duplicating it.
  • Cloud model quota. During the build, our brand-new cloud account had a model quota of zero, so every hosted model call was refused. Instead of stalling, we built the fast local fallback. The demo video was recorded in that state: the same agent runs on a local model, and the video says so.
  • macOS accessibility is slow on big windows. We replaced a deep entire contents walk with a bounded breadth-first walk (depth, node and time budgets), falling back to vision when a target has no accessible label.

Accomplishments that we're proud of

  • An agent that produces useful, evidence-backed output from nothing but app switches. On a synthetic bookkeeper week it finds the Gmail → Google Sheets → Pennylane loop (11 occurrences over 5 days) and ignores Slack, Spotify and YouTube noise.
  • Privacy by construction: behaviour only, redaction before any model call, pause, erase and forget.
  • One agent codebase that runs locally or in the cloud, and degrades gracefully when a model is unavailable.

What we learned

  • The best way to make an LLM agent trustworthy is to give it tools that return counted evidence, and to make its write tools refuse anything that isn't backed by that evidence.
  • A model-driven agent loop makes it natural to split the work: deterministic code does the counting, the model does the judging.
  • Plan for quotas and provider failures from day one: detecting them fast matters as much as falling back.

What's next for Cofounder

  • Cross-device memory of workflows and facts.
  • Correct a detected workflow in one sentence, and replay the correction into its mission.
  • Scheduled takeovers ("every morning at 9"), and team-level detection of workflows shared across an office.
  • A signed macOS build for anyone to install.

Pre-existing code and third-party disclosure

[Adapt this section to this hackathon's rules on the building period and pre-existing code.]

The Cofounder project started on 10 August 2026. It reuses parts of an earlier prototype by the same author (the Mira desktop app, June–July 2026): the Electron window and onboarding UI design, the nut.js input bridge, and the skeleton of the agent orchestrator (which was Windows-only and used a local Ollama model). Everything related to observation, workflow detection, memory, the workflow agent, the macOS executor and the proactive UI was built from mid-September 2026. Open-source dependencies are listed in package.json and strands-agent/pyproject.toml. The code is released under the MIT License.

Built With

Share this project:

Updates

Submission history