Thursday Activated
Thursday In Motion

Thursday (T.H.U.R.S.D.A.Y.)—Technologically Helpful Universal Reasoning System for Digital Accessibility and You

Inspiration

Computers are incredibly capable, but using them still requires learning the computer's language. You need to know which settings menu contains the option you need, what an unfamiliar icon means, where a website hid a particular button, or what sequence of steps will solve a problem.

That is inconvenient for everyone, but it can become a genuine barrier for people with visual, motor, or cognitive accessibility needs and for people who simply aren't comfortable with technology.

We wanted to reverse that relationship. Instead of requiring the user to adapt to the computer, Thursday makes the computer adapt to the user.

What it does

Thursday is an accessibility-first AI agent that can understand and interact with a Windows (MacOS coming soon!) computer.

Rather than relying entirely on screenshots, Thursday builds a structured semantic understanding of the computer's interface. It can inspect applications and UI elements, search them by meaning, reason about what the user is trying to accomplish, and perform actions through Windows accessibility and system interfaces.

A user can ask Thursday things such as:

“What’s on my screen?”
“Make everything bigger.”
“Open my downloads.”
“Where are my recent orders?”
“Why can’t I hear anything?”
“What does this window want me to do?”

Thursday can then reason about the request, retrieve relevant interface elements, call the appropriate computer-control tools, observe what changed, and continue until the task is complete.

The goal isn't just another AI that can move a mouse. Thursday is designed as an accessibility layer between a person and their computer, allowing interaction through intent rather than requiring the user to understand the mechanics of every interface.

How we (and our agents) made it

We split Thursday into three largely independent components:

The interface is built with Tauri 2 and provides the user's interaction with Thursday.

The intelligence core is a Python service responsible for AI reasoning, tool orchestration, context management, safety decisions, memory, and semantic retrieval using the Gemini API and

The hardware bridge translates Thursday's platform-independent actions into actual Windows operations using accessibility APIs, system APIs, and input control.

The components communicate locally using a versioned JSON protocol over WebSockets.

A major part of our architecture is Thursday's semantic world model. Windows UI elements are represented as structured objects containing information such as their role, name, description, location, visibility, and relationships to other elements. Thursday maintains a semantic index over this information so that natural language doesn't have to exactly match the text on screen.

For example, a request like “show me the stuff I bought recently” can semantically resolve to a control labeled “Returns & Orders.”

Structured interface information is our preferred source of truth. Screenshots and vision can be used when necessary, but Thursday is designed to avoid repeatedly taking screenshots and asking an AI where to click when the operating system already exposes that information directly.

The intelligence core also separates short-term working context, persistent memory, and the live semantic representation of the computer. This gives the agent relevant context without requiring the entire interaction history or computer state to be sent to the model every time.

Challenges

One of our biggest challenges was defining the boundary between an AI agent and the operating system.

Computer interfaces change constantly. A button can move, disappear, become disabled, or cause an entirely new window to appear between two actions. We needed Thursday to observe state, act, and then re-evaluate rather than blindly execute a long sequence against stale information.

We also had to coordinate three independently developed parts of the project. We created a shared protocol and tool catalog so the UI, intelligence core, and Windows bridge could be developed against mocks before all three pieces were connected.

Another challenge was deciding when Thursday should use structured accessibility information, direct system operations, simulated input, or visual understanding. Our solution was to treat these as progressively more invasive ways of accomplishing the same intent, preferring structured and direct approaches whenever possible.

Accomplishments that we're proud of

We're especially proud that Thursday became an actual end-to-end agent rather than just a concept or AI chat interface.

We built a common protocol that allows independently developed UI, intelligence, and operating-system components to work together; a tool-calling agent capable of performing multi-step computer tasks; a structured world model of the computer interface; semantic retrieval over UI elements; and safeguards around actions with different levels of consequence.

We're also proud of the architecture behind it. Thursday doesn't fundamentally depend on Windows-specific concepts. The intelligence layer operates on a common set of tools and semantic objects, meaning another platform adapter could eventually expose the same capabilities on macOS or Linux.

What's next

Our immediate goal is to expand the number and reliability of tasks Thursday can perform and improve its understanding of complex applications.

Longer-term, we'd like to add richer browser semantics, more sophisticated semantic indexing, better voice interaction, user-controlled long-term memory, and additional platform adapters.

We also want to explore Thursday as more than an agent that performs tasks for you. It could guide a user through a task visually, explain unfamiliar interfaces in plain language, generate simplified task-specific views of complicated applications, and act as a semantic screen reader that describes what an interface means rather than simply reading every element aloud.

Ultimately, we'd like interacting with a computer to require less knowledge about how the computer works and more freedom to simply say what you want to accomplish.