ST Engineering Hackathon - Big QYL

Inspiration

Security monitoring tools and cyber ranges need realistic desktop activity to train and test against — but most automation gives itself away. Scripts click too fast, type at a fixed rhythm, and spawn tell-tale processes that light up instantly in Sysmon or the Windows Event Log. We wanted to build an agent that doesn't just complete office tasks, but completes them in a way that looks human all the way down to the process tree.

What it does

Big QYL is a desktop automation agent that runs two realistic office workflows inside a Windows VM:

Office Secretary — writes meeting notes in OpenOffice Writer, updates salaries and reimbursements in a Calc profit-and-loss workbook, browses Apple and Amazon for purchasing, and finalises the spreadsheet with live formulas.
Financial Researcher — browses CoinMarketCap and AP News, takes research notes, then drafts a financial report in a side-by-side split-screen layout.

It drives real applications through their actual interfaces, and the whole run is designed to be indistinguishable from a human when viewed through system-monitoring tools.

How we built it

Everything runs from a single orchestrator (orchestrator.py) written in Python with PyAutoGUI and the Windows API (ctypes). Key design choices:

Single process — the orchestrator imports each teammate's module instead of launching subprocesses, so the entire demo runs inside one pythonw.exe with no console window and no child interpreters. Data passes between stages in memory, leaving no files on disk.
Human-like input — log-normal timing between actions, cubic Bézier mouse paths that accelerate and settle, variable typing speed with occasional adjacent-key typos and backspace corrections.
Clean launch — the agent is started via Windows Task Scheduler, and apps are opened through the shell, so their parent process is explorer.exe — exactly like a real user.
Adaptive pacing — the orchestrator targets a ~15-minute run by adjusting only the pauses between stages, and a Shift+Esc watcher thread allows an instant emergency stop.

Challenges we ran into

Keeping telemetry clean — our first builds left a debug log on disk and ran with a visible console. We switched to pythonw, in-memory handoffs, and shell-based launches to erase those tells.
Integrating three people's code — browsing, Writer/Calc, and orchestration were built separately. Merging them into one process without breaking each part took careful interface design.
Coordinate fragility — GUI automation depends on screen resolution and window layout, so a lot of tuning went into keeping clicks landing on the right cells.
Isolated-network SSL warnings — the VM's certificate interstitials broke browsing until we added title-based detection and a keyboard-driven bypass.
Recording headaches — the VM kept locking mid-run; we had to disable sleep/screensaver and record entirely inside the VM.

Accomplishments that we're proud of

A full run completes both scenarios in ~16 minutes with 0 failed stages.
The telemetry holds up: applications launch with parent explorer.exe, the agent runs as a single pythonw.exe (~0.5% CPU, ~11 MB RAM), and no cmd, conhost, powershell, or debug files are ever created.
We verified all of this ourselves with Sysmon, Process Explorer, and Task Manager.

What we learned

How ordinary automation exposes itself at the process and event-log level — parent processes, console windows, and stray files all matter.
Practical Sysmon and Windows process telemetry, and behaviour-tree-style control flow for coordinating tasks.
How to split a project across a team and still ship it as one clean, single-process program.

What's next for Big QYL

Replace fixed coordinates with UI-element lookup (accessibility APIs / image matching) so it survives resolution changes.
Add on-screen verification so the agent can detect and recover from unexpected dialogs.
Support more applications and more data-driven content that reflects what was actually browsed.