Windows Harness

robot
architecture

Inspiration

The most annoying part of working with agents for coding is testing the UI work after they are done. Playwright exists which helps a lot with browser apps. But I needed something for windows UI.

What it does

For the CI/CD integration it maintains scripts for previous UI elements it has encountered. When a commit touches new UI elements the agent explores the changes and writes new automated UI tests for any Windows UI. Works on Qt on Window's, all native Window's frameworks, DOM based UI, and everything else through point at X,Y fallback. All deployed to google cloud.

My experience with full app automation across modals is that it is very flaky. By having the agent lock in deterministic scripts over passes it resolves a lot of these issues.

The core tool is very operable across workloads. It is a py lib for agents to write automations in and interact with UI in real time.

How we built it

Python and UIA. Used google cloud servers to host a windows VM to test on and deploy ci/cd to.

Challenges we ran into

UIA provides windows UI automation for most window's frameworks. To capture all other possible UI frameworks I had to build fall back layers for DOM, standard py UI automation pylibs.

Accomplishments that we're proud of

I got an agent to play Runescape during testing. It ain't tick manipulating, but it can (sometimes) click a cow.

What we learned

This basically does not work as an MCP server. The agent must iterate on a python script to properly navigate UI, which is good for reproducing actions.

https://youtu.be/ryKm-AVvag8