Cofounder

Show a workflow once. An agent takes it over and checks every step with OpenCV 5, on AWS Lambda (Graviton). It asks a human whenever the screen does something the demonstration didn't.

Comment

Inspiration

Office work is full of short, repetitive screen workflows: open an invoice, match it to the purchase order, approve it, thirty times a morning. Automation tools start with a spec nobody writes. Click-replay bots break as soon as a window moves or a dialog appears, and they fail silently.

What it does

  • Learns from one recording. OpenCV 5 turns a screen recording into steps:
    • frame differencing and histograms find the stable screens;
    • masked template matching tracks the cursor;
    • a dwell followed by a local change marks a click;
    • each click keeps a crop of the clicked control.
  • Sees before acting. It finds each control again with multi-scale normalized cross-correlation on gray and Canny edges, with an ORB + RANSAC homography fallback and an ambiguity test. When there are several identical buttons, ordinal grounding picks the same one as in the demonstration, e.g. "the 1st of 5 Review buttons".
  • Checks after acting. verify_change (absdiff, morphology, SSIM) returns changed, no change or unexpected. It also compares the effect with the one the demonstration produced.
  • The visual evidence changes what the agent does next:
    • no change: wait and re-check before any retry;
    • unexpected dialog: ask the human;
    • nothing found: look again, then escalate.
  • Guards live in the tools. click() refuses any coordinate OpenCV did not return. Irreversible actions need a batch approval from a human.
  • Every decision is traced. Each look, score, verdict and human answer goes to a trace, with annotated frames.

How we built it

  • OpenCV library: cofounder_vision, OpenCV 5.0.0 (opencv-python-headless==5.0.0.93, numpy pinned), no contrib modules.
  • On AWS: served from AWS Lambda arm64 (Graviton) through a public Function URL that also hosts the web demo. Uploads go to S3 through presigned URLs, are deleted after processing and expire within 24 h. CloudWatch receives structured JSON logs. Deployment is idempotent (boto3), and a smoke test runs 28 checks.
  • The agent: Strands Agents with a local model (Ollama gemma4), or a deterministic policy baseline. Both use the same five tools.
  • Evaluation target: Ledgerly, an accounts-payable demo app the agent sees only as pixels, with fault injection.

Challenges we ran into

  • A security dialog's blue button matched the "Review" template (score 0.86), and the agent clicked it. We fixed it in two ways: the agent now compares each click's visual effect with the effect seen in the demonstration, and the threshold for acting is now 0.90.
  • A small local LLM lost track over 15 steps. Per-step conversations with a tool budget fixed it in the normal case.
  • Amazon Bedrock was not available on our account, so the LLM runs locally.

Accomplishments

Results are measured by the scripts in the repo.

  • Perception benchmark:
    • locate hits 82.6–83.1 % over 480 perturbed runs (100 % for translation and 0.8–2.0× scale), with 2.1 % false positives;
    • verify_change accuracy is 97.0 % on 168 pairs;
    • segmentation finds 21 of 21 ground-truth steps, with 95.5 % precision.
  • Agent with the policy brain (perception on AWS):
    • 6 of 6 invoices approved in the clean, 125 %-zoom and slow-app scenarios, with no wrong click;
    • it stops and asks when a button never enables.
  • Failures are reported too, including dialog runs that escalate early and the LLM's misses.

What we learned

An agent becomes trustworthy when its tools return counted visual evidence, and when its action tools refuse anything that evidence doesn't support.

What's next

  • Typed text (OCR or a VLM) for steps that need input.
  • Real-screen recordings in the benchmark.
  • Bedrock models when access is granted.
  • A "Show me once" button in the Cofounder desktop app.

Built With

Share this project:

Updates

Submission history