Team 20: Mimic Agent

A single Windows executable that spends ~15 minutes doing ordinary office work while producing telemetry that looks like a real person at the keyboard at the process and Sysmon level.

Comment

1. The problem it solves

The reference design is CMU SEI's GHOSTS framework. GHOSTS is powerful but heavy, and more importantly, its browser automation is easy to detect:

  • it drives Chrome through Selenium, which spawns chromedriver.exe;
  • the browser launches with --enable-automation and a remote-debugging port;
  • automated apps are parented under the automation harness, not under explorer.exe.

Any one of those is a dead giveaway in a process tree. Therefore, we decided to not simulate a user controlling the machine, and actually control the machine the way a user does.

That means genuine OS input (SendInput) into real shell-launched applications, with human-like timing throughout. That includes keystrokes, mouse moves, and scroll wheels going into real windows.

2. Design principles

  • Look human in the process tree: Apps are launched through Explorer so they parent under explorer.exe. No chromedriver, no cmd/powershell parents, no automation flags on the command line.
  • Use real input: Every keystroke and mouse move goes through the Win32 SendInput API, which is the same path a physical keyboard/mouse uses.
  • Be non-periodic: Delays are drawn from heavy-tailed, two-state (active/thinking) distributions so there is no detectable rhythm.
  • Make mistakes: Typing injects occasional typos and corrects them with backspace.
  • Move like a hand: The mouse follows a curved, momentum-overshoot path (WindMouse) with idle fidgets, not teleports or straight lines.
  • Stay safe & reversible: The agent only touches things it created. It kills only the exact PIDs it started, has an emergency-stop hotkey, and self-audits its own timing.

3. The core technique

3.1 Launching so the parent is explorer.exe

A naive Shell.Application.ShellExecute parents the new app under the agent, which we do not want. Instead, AppLauncher uses the ExecInExplorer COM pattern and reaches into the running Explorer desktop and asks it to do the launch, so the new process parents under explorer.exe just like a user's double-click.

The chain is:

ShellWindows → FindWindowSW(CSIDL_DESKTOP) → IServiceProvider
   → IShellBrowser → IShellView.GetItemObject(SVGIO_BACKGROUND)
   → IShellFolderViewDual.Application → IShellDispatch2.ShellExecute(exe, args)

Then it discovers the new PID by diffing the process list before/after, waits for the window, and takes foreground with an AttachThreadInput guard so SetForegroundWindow reliably succeeds.

3.2 Genuine keystrokes and mouse via SendInput

NativeInput wraps the Win32 SendInput API with the correct x64 INPUT union layout (40-byte struct). Keystrokes go out as Unicode events (KEYEVENTF_UNICODE), so any character types correctly regardless of layout; Ctrl/Alt combos use virtual-key events. Mouse moves are sent as absolute, virtual-desktop-normalized coordinates (0–65535 across all monitors).

Because these are the same events a physical device produces, nothing downstream can tell them apart from a human — there is no automation surface to detect.

4. Making it look human

4.1 Timing — DelayModel

All waits come from a model and is never a constant:

  • Two-state Markov process: the agent flips between an active state (quick, tight delays) and a thinking state (longer pauses) like how a person alternates between typing and reading.
  • Log-normal, heavy-tailed delays: most actions are fast, but a long tail of occasional long pauses appears.
  • Keystroke latency depends on the character pair (e.g. same-hand vs. alternating), so typing cadence itself is non-uniform.

4.2 Typing — HumanTyping

Types text character-by-character with model latencies, and injects occasional typos that it then corrects with backspace.

4.3 Mouse — HumanMouse

  • WindMouse algorithm plans a curved path with simulated gravity + wind, so the cursor arcs toward the target and slightly overshoots with momentum, then settles.
  • Fitts's law sets the movement duration from distance and target size.
  • A fast initial flick plus small idle Wiggle fidgets during reading/scrolling.
  • Every destination is clamped to the virtual screen bounds.

5. The activity handlers

5.1 OpenOffice (OpenOfficeHandlers.cs)

The range runs Apache OpenOffice 4.1.16.

  • Discovery: finds ...\OpenOffice 4\program\soffice.exe. The live process image is soffice.bin.
  • Valid documents from scratch: MinimalOdf writes genuine ODF packages. Writer's .odt needs a real styles.xml with a page-layout + master page or OpenOffice shows the Start Center instead of the document.
  • Launch with -norestore (and -writer/-calc) so a prior crash's restore dialog can't interfere, then focus by window title ("OpenOffice Writer/Calc") with retries.
  • Writer handler: opens a pre-made .odt, types a short memo, saves silently with Ctrl+S (opening a pre-made file avoids the Save-As path-picker dialog that the agent used to get stuck in).
  • Calc handler: types a "Weekly Produce Inventory" table and a =SUM() total row.
  • Human cell navigation: per the range's preference, the agent walks cell-to-cell with arrow keys (Right across a row, Down + Left back to column A) rather than jumping.

5.2 Browser (BrowserHandler.cs)

The real browser is shell-launched (explorer-parented) and driven entirely with real input:

  1. Isolate the instance: snapshot browser PIDs before launch, then treat every browser PID that appears afterward as "ours".
  2. Random subset each run: from a pool of ~19 allowed sites, it picks a random 2–4 to visit.
  3. Navigate like a person: Ctrl+L to focus the address bar, type the URL, Enter.
  4. TLS-intercept interstitial: the range proxies everything through a TLS-intercepting proxy, so each new host first shows "your connection isn't private." The handler detects that by title and clicks Advanced → Proceed.
  5. Read: human-burst scrolling with idle wiggles and the occasional scroll-back-up.
  6. Follow a real content link: links are gathered from the page Document, restricted to the content band (skipping the top ~22% header/hero and the footer), filtered against a chrome/consent keyword blocklist, required to be ≥12 characters, de-duplicated, and biased toward the longest (headline) text.
  7. Dead-link skip: after clicking, it waits for the window title to actually change. If the title doesn't change (e.g. the click hit a dead region), it logs "link did nothing — moving on" and proceeds to the next site instead of getting stuck.
  8. Go two layers deep sometimes: on news/shopping-type sites it will often (not always) follow a second link for a more realistic reading session.

6. Orchestration, safety, and cleanup

6.1 SessionRunner

A lean orchestrator (not a full config engine). It builds the activity list (oo-calc, oo-writer, browser), shuffles the order, and cycles through them with realistic idle gaps (mostly 6–25 s, occasionally ~45 s) until the time budget (default 15 min) is spent. It logs the start/end of each activity and cleans up that activity's processes before moving on.

6.2 Emergency stop & process registry (Control/Session.cs)

  • A 50 ms background watcher listens for Ctrl+Alt+Shift+Q. On press it throws a stop exception that unwinds the run and closes anything still open.
  • A launched-process registry records every PID the agent starts, so an abort can guarantee nothing is left running.

6.3 Identity-checked cleanup

Cleanup never does taskkill /IM soffice.bin (that would kill the user's apps too). It kills only the exact PIDs the agent launched, and KillIfMatches re-verifies the image name of that PID before killing, so even if the OS recycled the PID, the agent won't kill an unrelated process.

7. What makes it "indistinguishable"

Put together, at the telemetry level a monitoring stack sees:

  • soffice.bin and chrome.exe parented under explorer.exe, launched like a double-click.
  • no chromedriver.exe, no --enable-automation, no remote-debug port, no cmd/powershell wrappers.
  • keyboard and mouse input arriving through the normal Win32 input path.
  • non-periodic action timing with a human heavy tail.
  • real documents saved and real (allowed) sites browsed, with mistakes, pauses, scrolling, and dead-end clicks.

8. Result

A single .NET 9 project, net9.0-windows, published as a self-contained single-file WinExe. It drops onto a VM desktop and runs on a double-click.

Challenges faced

The main challenge faced was not knowing where to start. Our team has not worked on a project like this before. However, after researching online and coming across GHOSTS, we had a better idea of the solution.

Another challenge was dealing with web browsing. There are many things to consider when doing web browsing, such as which links to click, what happens if you clicked on the wrong link, and even how to find the link in the first place. There was this time where the agent will click on 'Network Diagnosis' which will open the Windows Network Diagnosis popup and cause trouble for the agent.

Future improvements

With more time, we believe that it would be a good idea to frame the actions around a story. For example, the story could be 'Sorting food inventory', and during the 15 minutes, the agent will be carrying out actions regarding that. The agent could be doing inventory on the spreadsheet, making notes about the inventory on the document, and even browse the web to buy new inventory from Amazon.

Furthermore, we would have also liked to explore opening more applications with the agents and doing more with it to see how far it could go.

Acknowledgements

  • cmu-sei/GHOSTS
  • Claude Code

Built With

Share this project:

Updates

Submission history