It is a Python agent that carries out a session of up to fifteen minutes through the visible interface, with a real mouse pointer and real key presses.
Browses in Chrome. It reads homepages from a shuffled list of 27 sites, with each address typed at most once per run. On certificate warnings it clicks Advanced, then Proceed, and skips sites with no Proceed link. It accepts consent notices and closes other pop-ups with their X. It runs a find-in-page search on at most two sites, goes back to an earlier tab at most twice, and keeps to three tabs at most. Builds a workbook in OpenOffice Calc. It builds an eight-column, seven-record stock-inventory table with formulas, two corrections, a totals block, a bold header, currency format and a sort by Item. It saves through the Windows Save As dialog. Writes notes in Notepad. It types a short account of the session, ending with the line END. The notes stay on screen, unsaved, for whoever comes back to the computer. How we built it Three task agents on a shared core. An orchestrator (main.py) runs the browser, spreadsheet and notes agents in turn. They share a human-input layer (gui_support.py) and a pure-logic module (domain.py). The code is 4,872 lines of Python using pyautogui, pywinauto and pywin32. Process authenticity. The agent starts from a desktop shortcut running pythonw.exe, so Explorer is its parent and no console appears. It never calls subprocess. It shows the desktop and double-clicks the Chrome, Calc and Notepad icons, so Explorer starts each of them. It writes no files during a run: no logs, no report, no history and no Python cache. Only Calc writes the workbook. Human timing and movement. We built the models from a 223-second recording of a person browsing by hand, plus our own VM measurements. Pointer travel follows Fitts’ law with lognormal noise, a minimum-jerk path, a shallow arc, tremor and overshoot. Typing has per-key gaps, held keys, Shift timing and corrected slips. Pauses are lognormal. A new “personality” (pace, wheel speed, typo rate and pause length) is drawn for every run. pyautogui’s built-in pause is switched off, so there are no fixed sleeps. Reading Calc by eye. OpenOffice Calc exposes no cells, Name Box or menus to UI Automation. The agent reads the grid from small screen rectangles, using the highlighted headers and separator lines. It clicks cells, drags ranges and checks the selection from the headers. A 15-minute budget. Browsing gets up to 4:30, the spreadsheet 8:00 plus any unused browsing time, and the notes the rest. Five checkpoints compare progress with pilot timings. If the run is behind, it shortens pauses and skips optional steps. The notes choose lines by priority, so a minor line is dropped before an important one is cut. Light on resources. Waits are event-driven. Accessibility-tree walks are bounded, the screen is read in small rectangles, and there are no background threads. Testing without the VM. We have 424 automated tests that run on any computer, including a fake Chrome and a simulated OpenOffice Calc. The simulation checks that the workbook finishes within eight minutes even on a VM 1.3 times slower. Challenges we ran into
We couldn’t run the VM from our laptops. Every test run was screen-recorded and analysed frame by frame, and each of the seven rounds produced fixes and new tests.
Calc. Range selection never worked in round 1, so everything was retyped. Ctrl+D opens a Selection List instead of filling down, and Tab after a paste overwrote the pasted cell. We moved to mouse-led work with the grid read from pixels, and to copy and paste. The Save dialog. The workbook was saved in the wrong folder when the left pane was scrolled, and later in “This PC”. The agent now reads the address bar, scrolls the pane until Desktop is in view, types the full path rather than saving elsewhere, and answers No to any “replace?” question outside the agent folder. A bad copy into the VM. One wrongly copied file crashed a run 4.5 minutes in. The launcher now checks every module against a BUILD.json hash manifest before any input. Logging. A teammate’s Sysmon review showed pythonw.exe writing logs every few seconds. We removed all log, report, history and cache files. Timing. A Full demo left the totals unfinished, so the spreadsheet now has priority, with a hurry mode. Notes were cut mid-line when typing ran slower than planned, so the notes now re-choose their lines before each one and always end with END. Searching on every site and a temporary row cost time, so we limited searches to at most two sites and removed the row. Messy pages. CNN’s consent pop-up and Bitly’s offer pop-up both got in the way. Some pages render blank or turn white once scrolled. The agent checks that a warning names the host it asked for, so a leftover warning from the previous site isn’t acted on. Host interference. The host’s mouse once nudged the VM pointer. The agent now checks the pointer before each press. Accomplishments that we’re proud of
The final run was a Full demo on 17 October 2026, started at 9:46 PM and checked against its screen recording frame by frame (13,311 frames).
It finished inside the limit. It took 14 min 15 s from Start to the last letter of END, against a 15-minute limit. All three tasks completed. It read 4 homepages and skipped asana.com because its warning had no Proceed link. It ran one find-in-page search (“Asia” on CNN) and made one return to an earlier tab. It never had more than three tabs open. The notes had six lines, then END. The workbook is correct. inventory_oct17.ods has 7 records, formulas, two corrections and a total of SGD 3,486, within the SGD 5,000 budget. It is sorted by Item and saved in the agent folder. The trace is clean. Every application was launched by Explorer from its desktop icon. The workbook was the only item in the agent folder modified during the run. It stayed light. The agent process averaged about 0.11% CPU, with a peak of 0.86%, and about 60 MB in earlier VM measurements. The whole VM averaged 8.9% of its 4 vCPUs from Start to END (median 4.2%). The VM was under 10% for 69% of the run and under 20% for 89%. Its highest reading was 85.3%, as Chrome started. Memory stayed between 1.95 and 3.31 GiB of 8 GiB. We read these figures from every frame of the Proxmox panel in the recording. What we learned A smooth screen doesn’t prove authentic telemetry. We checked the trace and the recording, not just the screen. Human-like behaviour comes from distributions and habits (reading a warning before clicking, copy and paste, saving after milestones, trying a what-if value and putting it back), not from slower fixed delays. Without log files you diagnose from screen recordings and the result box. That is the price of a clean Sysmon trace. Peaks in CPU came with something starting or loading, and passed within seconds. Reading and typing stayed light. The figures in Section 10.3 of the report measure the whole VM, not the agent alone. They include Windows, the applications and the console stream.
What’s next for Synthetic human using python Detect a blank render on screen and move on sooner, which would leave time for one more site.
If possible add context with how we gonna design the product i.e make it do a specific workflow instead of doing independent task
Log in or sign up for Devpost to join the conversation.