Inspiration
The idea for ScreenPilot came from my uncle, who has a slight handicap and struggles with modern technology. Many apps and websites assume a certain level of digital literacy, which can make even simple tasks like booking something online, sending an email, or navigating settings surprisingly difficult.
Watching him struggle with interfaces that feel intuitive to most people made me realize how much of today’s software is not truly accessible. I wanted to build something that removes the complexity of interacting with technology. Instead of forcing people to learn interfaces, the interface should adapt to the user.
That’s where the idea came from: what if someone could simply tell their computer what they want to do, and the computer handles the rest?
What it does
ScreenPilot is an AI agent that can operate a computer on behalf of the user.
The user simply speaks or types a goal, such as:
“Open my email and send a message to John.”
“Find a cheap flight to London next week.”
“Change my display brightness.”
The agent then:
Captures the user’s screen
Uses Gemini multimodal AI to understand what is visible on the interface
Determines the next step needed to reach the goal
Executes mouse and keyboard actions automatically
The process repeats step-by-step until the task is completed.
In practice, this allows users who struggle with technology to interact with their computer using natural language instead of complex interfaces.
How we built it
We built ScreenPilot by combining several components:
Screen capture to continuously observe what is happening on the user’s screen
Gemini multimodal to analyze screenshots and understand UI elements such as buttons, menus, and text fields
A lightweight agent loop that plans the next action needed to complete the user’s goal
Mouse and keyboard automation to execute actions on the computer
A simple interface where users can type or speak instructions
We relied heavily on experimentation, critical thinking, and AI-assisted development tools like Claude to rapidly prototype and refine the system.
The core idea was to build a feedback loop:
Observe the screen
Reason about the interface
Take an action
Observe again
This loop allows the agent to gradually work its way toward completing the user’s request.
Challenges we ran into
One of the biggest challenges was deployment, since this was my first time deploying a project like this. Getting all the pieces to work together outside the development environment required a lot of troubleshooting.
Another challenge was reliability. Interfaces change, buttons move, and layouts are not always consistent. Teaching an AI agent to robustly interpret these variations was more complex than expected.
We also had to think carefully about safety, making sure the agent doesn’t perform unintended actions or lose track of the user’s goal.
Accomplishments that we're proud of
One thing we are particularly proud of is building a working end-to-end agent that can observe a screen, reason about it, and perform real actions.
We’re also proud that the project focuses on accessibility rather than just automation. The goal is not just convenience, but helping people who struggle with traditional interfaces.
Finally, we managed to go from idea to a functional prototype relatively quickly, proving that this approach is technically possible.
What we learned
Through this project we learned a lot about:
Building AI agents that interact with real environments
Working with multimodal models
Designing automation systems that interact with unpredictable user interfaces
Deploying and packaging a full project for others to use
We also learned that accessibility problems are often not about ability, but about how software is designed.
What's next for ScreenPilot Agent
There are many directions we want to explore next.
First, we want to improve reliability and safety, ensuring the agent can consistently complete tasks without mistakes.
We also want to add:
Voice interaction for fully hands-free control
Better planning and memory, allowing the agent to handle longer workflows
User confirmation steps for sensitive actions
In the long term, the vision is to create an assistant that allows anyone — regardless of technical skill — to control their computer simply by describing what they want to do.
Log in or sign up for Devpost to join the conversation.