Synthetic Human Using Python

Inspiration

Making a computer look busy is easy. Making it look like a real person is using it is much harder. A typical automation script gives itself away: apps launched from a console, fixed sleeps between actions, perfectly straight mouse paths, and log files no human would ever write.

For the NTU × ST Engineering Hackathon, Building a Synthetic Human Agent, we set out to build an agent that holds up at two levels. On screen, it moves, types, hesitates and makes mistakes like a person. In the system trace, every process, launch and file write looks like an ordinary user session.

In our final run, the agent completed a full 15-minute work session (browsing, a spreadsheet and notes) in 14 min 15 s, with a clean trace and the whole VM averaging under 9% CPU.

What it does

Our agent is a Python program that carries out a session of up to fifteen minutes entirely through the visible interface, using a real mouse pointer and real key presses.

Browses in Chrome

Reads homepages from a shuffled list of 27 sites, typing each address at most once per run.
On certificate warnings, clicks Advanced, then Proceed, and skips sites with no Proceed link.
Accepts consent notices and closes other pop-ups with their X.
Runs a find-in-page search on at most two sites, returns to an earlier tab at most twice, and never keeps more than three tabs open.

Builds a workbook in OpenOffice Calc

Creates an eight-column, seven-record stock-inventory table with formulas, two corrections, a totals block, a bold header, currency formatting and a sort by Item.
Saves through the Windows Save As dialog.

Writes notes in Notepad

Types a short account of the session, ending with the line END.
Leaves the notes on screen, unsaved, for whoever comes back to the computer.

How we built it

Three task agents on a shared core. An orchestrator (main.py) runs the browser, spreadsheet and notes agents in turn. They share a human-input layer (gui_support.py) and a pure-logic module (domain.py). The code is 4,872 lines of Python using pyautogui, pywinauto and pywin32.

Process authenticity

The agent starts from a desktop shortcut running pythonw.exe, so Explorer is its parent and no console appears.
It never calls subprocess. It shows the desktop and double-clicks the Chrome, Calc and Notepad icons, so Explorer starts each of them.
It writes no files during a run: no logs, no report, no history and no Python cache. Only Calc writes the workbook.

Human timing and movement. We built our models from a 223-second recording of a person browsing by hand, plus our own VM measurements.

Pointer travel follows Fitts' law with lognormal noise, a minimum-jerk path, a shallow arc, tremor and overshoot.
Typing has per-key gaps, held keys, Shift timing and corrected slips.
Pauses are lognormal.
A new "personality" (pace, wheel speed, typo rate and pause length) is drawn for every run.
pyautogui's built-in pause is switched off, so there are no fixed sleeps.

Reading Calc by eye. OpenOffice Calc exposes no cells, Name Box or menus to UI Automation. The agent reads the grid from small screen rectangles, using the highlighted headers and separator lines. It clicks cells, drags ranges and confirms the selection from the headers.

A 15-minute budget. Browsing gets up to 4:30, the spreadsheet 8:00 plus any unused browsing time, and the notes the rest. Five checkpoints compare progress against pilot timings; if the run falls behind, it shortens pauses and skips optional steps. The notes choose lines by priority, so a minor line is dropped before an important one is cut.

Light on resources. Waits are event-driven, accessibility-tree walks are bounded, the screen is read in small rectangles, and there are no background threads.

Testing without the VM. We wrote 424 automated tests that run on any computer, including a fake Chrome and a simulated OpenOffice Calc. The simulation checks that the workbook finishes within eight minutes even on a VM 1.3× slower.

Challenges we ran into

No VM access from our laptops. Every test run was screen-recorded and analysed frame by frame, and each of the seven rounds produced fixes and new tests.

Calc. In round 1, range selection never worked, so everything was retyped. Ctrl+D opened a Selection List instead of filling down, and Tab after a paste overwrote the pasted cell. We moved to mouse-led work, with the grid read from pixels, and to copy and paste.

The Save dialog. The workbook was saved in the wrong folder when the left pane was scrolled, and later in "This PC". The agent now reads the address bar, scrolls the pane until Desktop is in view, types the full path rather than saving elsewhere, and answers No to any "replace?" prompt outside the agent folder.

A bad copy into the VM. One wrongly copied file crashed a run 4.5 minutes in. The launcher now checks every module against a BUILD.json hash manifest before any input.

Logging. A teammate's Sysmon review showed pythonw.exe writing logs every few seconds. We removed all log, report, history and cache files.

Timing

A Full demo left the totals unfinished, so the spreadsheet now has priority, with a hurry mode.
Notes were cut mid-line when typing ran slower than planned, so the notes now re-choose their lines before each one and always end with END.
Searching on every site and a temporary row cost time, so we limited searches to at most two sites and removed the row.

Messy pages

CNN's consent pop-up and Bitly's offer pop-up both got in the way.
Some pages render blank or turn white once scrolled.
The agent checks that a warning names the host it asked for, so a leftover warning from the previous site isn't acted on.

Host interference. The host's mouse once nudged the VM pointer. The agent now checks the pointer position before each press.

Accomplishments that we're proud of

Our final run was a Full demo on 17 October 2026, started at 9:46 PM and checked against its screen recording frame by frame (13,311 frames).

It finished inside the limit: 14 min 15 s from Start to the last letter of END, against a 15-minute limit.

All three tasks completed

Read 4 homepages and skipped asana.com because its warning had no Proceed link.
Ran one find-in-page search ("Asia" on CNN) and made one return to an earlier tab.
Never had more than three tabs open.
Wrote six lines of notes, then END.

The workbook is correct. inventory_oct17.ods has 7 records, formulas, two corrections and a total of SGD 3,486, within the SGD 5,000 budget. It is sorted by Item and saved in the agent folder.

The trace is clean

Every application was launched by Explorer from its desktop icon.
The workbook was the only item in the agent folder modified during the run.

It stayed light

The agent process averaged about 0.11% CPU, with a peak of 0.86%, and used about 60 MB in earlier VM measurements.
The whole VM averaged 8.9% of its 4 vCPUs from Start to END (median 4.2%).
The VM was under 10% for 69% of the run and under 20% for 89%. Its highest reading was 85.3%, as Chrome started.
Memory stayed between 1.95 and 3.31 GiB of 8 GiB.
We read these figures from every frame of the Proxmox panel in the recording.

What we learned

A smooth screen doesn't prove authentic telemetry. We checked the trace and the recording, not just the screen.
Human-like behaviour comes from distributions and habits, such as reading a warning before clicking, copying and pasting, saving after milestones, or trying a what-if value and putting it back, not from slower fixed delays.
No log files means diagnosing from screen recordings and the result box. That is the price of a clean Sysmon trace.
CPU peaks came with something starting or loading and passed within seconds. Reading and typing stayed light.
The figures in Section 10.3 of the report measure the whole VM, not the agent alone. They include Windows, the applications and the console stream.

What's next for Synthetic Human Using Python

Detect blank renders sooner. Spotting a blank page on screen and moving on faster would leave time for one more site.
One connected workflow instead of three separate tasks. Link the tasks into a single realistic job, for example looking up items online, recording them in the inventory workbook, then summarising the result in the notes, so the session reads as one person working toward one goal.

The new Inspiration section sets up the problem and gives the key results upfront. I also expanded your last "What's next" point into a concrete example; swap in whatever workflow your team actually has in mind.