Cofounder

An agent for your Mac that notices the workflows you repeat, remembers them precisely, and takes them over. Built with Strands Agents on Amazon Bedrock AgentCore.

Comment

About the project (story)

Inspiration

Every automation tool starts with the same question: "What do you want to automate?" We noticed that almost nobody can answer it. The work we repeat is invisible from the inside. A freelance bookkeeper opens an invoice in Gmail, copies it into a spreadsheet, and books it in her accounting tool, three times a morning, every morning. She would never call that "a workflow", and describing it precisely enough for a bot would take longer than just doing it. So the most repetitive work stays manual, not because it can't be automated, but because nobody ever writes the spec.

We wanted to flip the question around: an agent that watches how you work, notices what you repeat, and offers to take it over.

What it does

Cofounder is a macOS desktop agent with four layers.

  1. Observes behaviour, not content. Every two seconds, Cofounder records which app is in front, the window title and the browser tab's URL, and only when they change. It never records keystrokes, the clipboard or screenshots of your work. Observation is always visible and can be paused or erased in one click. E-mail addresses and long numbers in window titles are masked before any model sees them.
  2. Detects repeated workflows. Every 30 minutes, or when you click Analyse now, the last seven days of activity go to the Workflow Analyst, an agent built with the Strands Agents SDK and deployed on Amazon Bedrock AgentCore Runtime. A deterministic miner first extracts the app-and-site sequences that repeat. The Strands agent then does what a model is good at: it reads the evidence (timestamps, window titles), rejects navigation noise, checks what it already remembers, and saves only genuine workflows.
  3. Remembers precisely. Each workflow becomes a memory card: a human name, the ordered steps, the trigger, how often it happened, an honest estimate of the minutes it costs per week, a confidence score, the evidence it was built from, and an automation mission written for a computer-use agent. Durable facts about how you work ("books supplier invoices in Pennylane") are reconfirmed over time, never duplicated. Everything is listed in the app and can be dismissed.
  4. Speaks up and takes over. A macOS notification tells you when a new workflow is found. Take it over hands its mission to Cofounder's computer-use executor. The executor plans the steps with the model, reads the screen through the macOS accessibility tree (with a screenshot as a vision fallback), and drives the real mouse and keyboard. A visible overlay narrates every step, and there is a Stop button. Cofounder pauses its own observation while it works, so its clicks never count as your workflow.

How we built it

  • Strands Agents SDK (Python). One agent, one system prompt, and six @tool functions: get_activity_overview, find_repeated_workflows, get_workflow_evidence, recall_memory, save_workflow, remember_fact. The key design choice is that save_workflow must cite real candidate ids returned by the miner. The model can judge, name and explain workflows, but it cannot invent one the evidence does not show.
  • Deterministic mining (no model): it normalises and redacts events, splits them into work sessions at 15-minute gaps, collapses them into "stations" (an app, narrowed to a site for browsers), and removes alt-tab flicker. It then mines non-overlapping n-grams of distinct stations, drops sub-sequences explained by longer runs and rotations of the same loop, and scores by repetitions × length × distinct days.
  • Amazon Bedrock AgentCore Runtime. The agent is a BedrockAgentCoreApp entrypoint deployed with the AgentCore toolkit. The desktop app calls it with InvokeAgentRuntime (AWS SDK for JavaScript, SigV4). The activity travels in the payload, so the cloud agent never needs access to the user's disk.
  • Amazon Bedrock. Claude Sonnet 4.6 is the primary model, both for the Strands agent (BedrockModel) and for the desktop app (Converse API) when it chats, plans missions and uses vision.
  • Resilience. A one-shot Bedrock probe with no retries detects a refused or throttled account in about 2 seconds. The same Strands agent, tools and prompt then run on a local model through Strands' OllamaModel. If AgentCore can't reach a model, the app runs the agent locally. The app always reports which model provider answered.
  • Desktop app. Electron, React, Vite and Tailwind, with a System Events observer (AppleScript), a JSON memory store with atomic writes, a JXA accessibility-tree reader with strict time budgets, nut.js for input, and a transparent overlay window.

Challenges we ran into

  • Workflow vs. navigation. Our first miner output was dominated by alt-tab laps (A→B→A→B) and by rotations of the same loop (B→C→A is the same work as A→B→C): a dozen "candidates" for one real workflow. Requiring distinct stations and removing rotations and sub-sequences brought it down to a short list the model can actually judge.
  • Keeping the model honest. We did not want an agent that describes plausible-sounding workflows. So persistence depends on evidence ids returned by the miner, and a recall_memory tool lets the agent update what it knows instead of duplicating it.
  • A new AWS account starts with a Bedrock quota of zero. During the build, every Bedrock model call on our fresh account was throttled ("Too many tokens per day"), and the daily token quota is not self-service adjustable. We deployed to AgentCore anyway and built the fast local fallback so the product keeps working. The demo video was recorded in that state: the same Strands agent runs on a local model, and we say so in the video.
  • macOS accessibility is slow on big windows. We replaced a deep entire contents walk with a bounded breadth-first walk (depth, node and time budgets), falling back to vision when a target has no accessible label.

Accomplishments that we're proud of

  • An agent that produces useful, evidence-backed output from nothing but app switches. On a synthetic bookkeeper week it finds the Gmail → Google Sheets → Pennylane loop (11 occurrences over 5 days) and ignores Slack, Spotify and YouTube noise.
  • Privacy by construction: behaviour only, redaction before any model call, pause, erase and forget.
  • One agent codebase that runs locally, deploys unchanged to AgentCore Runtime, and degrades gracefully.

What we learned

  • The best way to make an LLM agent trustworthy is to give it tools that return counted evidence and to make its write tools refuse anything that isn't backed by that evidence.
  • Strands' model-driven loop made it natural to split the work: deterministic code does the counting, the model does the judging.
  • Plan for quotas and provider failures from day one: detecting them fast matters as much as falling back.

What's next for Cofounder

  • AgentCore Memory for cross-device recall of workflows and facts.
  • Correct a detected workflow in one sentence, and replay the correction into its mission.
  • Scheduled takeovers ("every morning at 9"), and team-level detection of workflows shared across an office.
  • Once the Bedrock quota is granted, run all detection on AgentCore with Claude by default.

Pre-existing code and third-party disclosure

The Cofounder project started on 10 August 2026. It reuses parts of an earlier prototype by the same author (the Mira desktop app, June–July 2026): the Electron window and onboarding UI design, the nut.js input bridge, and the skeleton of the agent orchestrator (which was Windows-only and used a local Ollama model). Everything related to observation, workflow detection, memory, the Strands agent, AgentCore deployment, Amazon Bedrock, the macOS executor and the proactive UI was built during the Submission Period. Open-source dependencies are listed in package.json and strands-agent/pyproject.toml. The code in the repository is released under the MIT License.

Built With

Share this project:

Updates

Submission history