ST Engineering Team 29

Inspiration

Many everyday office tasks involve repetitive interactions across multiple desktop applications. Creating spreadsheets, documenting meetings, and navigating websites can consume valuable time. We wanted to explore how a lightweight desktop automation agent could coordinate these workflows using ordinary graphical user interface interactions.

What it does

It automates a sequence of common desktop activities on a Windows virtual machine:

Spreadsheet automation: Opens Apache OpenOffice Calc, enters inventory and budget data, calculates totals using spreadsheet formulas, and saves the document.
Meeting documentation: Opens OpenOffice Writer, creates a structured meeting report, applies basic text formatting, adds action items, and saves the document.
Web navigation: Opens Google Chrome, visits three websites, scrolls through pages, interacts with page content, and navigates between pages.
GUI interaction: Uses visible cursor movement, mouse clicks, keyboard input, and standard application shortcuts.

How we built it

The prototype was developed in Python using standard libraries, including ctypes, random, time, and csv.

We used the Windows input API to implement keyboard and mouse interactions. The agent coordinates application opening, document creation, formatting, saving, and browser navigation through a predefined workflow.

The prototype runs on a Windows virtual machine with Apache OpenOffice and Google Chrome installed.

Challenges we ran into

One major challenge was making GUI automation reliable across different window positions and display settings. Mouse coordinates required calibration, and applications needed sufficient loading time before subsequent interactions.

We also had to account for differences between spreadsheet editing, word processing, and browser navigation. These applications require different keyboard shortcuts, input patterns, and timing considerations.

Accomplishments that we're proud of

One of our proudest accomplishments was developing a desktop automation agent that interacts with applications through visible mouse and keyboard actions, rather than relying solely on command-based execution.

Instead of launching applications through commands or keyboard shortcuts, our agent navigates the desktop, moves the cursor to application icons, and opens them through mouse clicks, closely resembling how a person would interact with a computer.

We also successfully integrated multiple applications into a single workflow, enabling the agent to create spreadsheets, generate formatted meeting reports, and browse websites. Additionally, we implemented variable typing speeds, occasional typing errors with corrections, and calibrated cursor movements to simulate different interaction patterns.

Most importantly, we transformed low-level Windows input operations into a functional, multi-application desktop automation prototype.

What we learned

We gained practical experience with Windows GUI automation, event-driven input, workflow orchestration, and application-specific interactions.

The project also highlighted the importance of error handling, timing calibration, and reliable state management in desktop automation.

What's next for ST Engineering Team 29

We hope to extend the prototype with application-state detection, more reliable visual element recognition, configurable workflows, and improved recovery from unexpected dialogs or application errors.