Synthetic Human Agent

Inspiration

We were inspired by a challenge in cybersecurity training: security tools need realistic everyday computer activity to evaluate their monitoring capabilities. An idle machine cannot represent the browsing, document editing, and application switching that happen during a normal workday.

We created Synthetic Human Agent to generate that background activity through a connected office workflow, while keeping its actions transparent and its telemetry auditable.

What it does

Synthetic Human Agent is a Windows desktop automation project designed around a 15-minute office session. It:

Launches a browser through File Explorer.
Browses public websites.
Opens Microsoft Word and types a report.
Returns to the browser for further browsing.
Finishes, saves, and verifies the document.

The agent uses curved mouse movements, variable typing intervals, reading pauses, occasional typing mistakes, and Backspace corrections.

It records actions and outcomes in audit logs, preserves browser automation indicators, and stops when it detects CAPTCHAs or access restrictions.

How we built it

We built the project in Python, with separate modules for scheduling, browser interaction, desktop input, Word document handling, and logging.

PyAutoGUI provides keyboard and mouse input.
Playwright supports browser observation and navigation checks.
Windows COM interfaces handle Word document setup, saving, and readback.
Bézier curves create curved cursor movements.
Randomized timing introduces variation into typing and reading pauses.
JSON configuration files define websites, session phases, and prewritten report text.

The report is typed character by character. At the end of a successful session, the agent saves and reopens the document, compares its contents with the intended text, and records a SHA-256 hash.

A Sysmon export helper supports examination of genuine Windows telemetry alongside the application logs.

Challenges we ran into

Switching between Word and Chrome

A window could appear on screen before it was ready to receive input. We reduced unnecessary window resizing and added focus checks and settling pauses before typing.

Missing characters in website addresses

Sometimes, https:// became ttps:// because the first character was lost during address entry. Chrome then performed a search instead of navigating.

We added address-bar readback to check the typed URL before pressing Enter and retype it when necessary.

Managing phase deadlines

An optional browsing action could exceed its phase budget and cancel the entire session before Word opened.

We separated browsing-phase expiration from the overall session deadline so optional browsing could end while the workflow continued.

Handling unpredictable websites

Public websites introduced redirects, cookie banners, popups, and regional selectors. Handling these interfaces while respecting access restrictions became an important part of the project.

Accomplishments we’re proud of

We connected browsing, document editing, application switching, and file verification into one coordinated workflow.

We also added:

Simulation mode for testing without controlling a desktop.
Cancellation controls and focus protection.
Detailed audit logs for investigating failures.
Document readback verification.
Clear distinctions between completed, blocked, and cancelled sessions.

What we learned

We learned that reliable desktop automation requires more than sending clicks and keystrokes. It must verify application focus, navigation results, and saved output.

We also learned that randomized behavior alone does not establish human realism. Meaningful tasks, connected activities, and accurate records are equally important.

Testing taught us to distinguish simulated success from real-world validation. Mocked tests help check logic, but Windows execution is necessary to confirm application behavior and timing.