Remote Control

Inspiration

Remote Control started with a very simple problem: being away from your computer is annoying.

As a developer, I constantly remember things I need to do when I'm nowhere near my computer. Maybe I'm on my phone, lying in bed, or somehow having my best idea in the bathroom. Instead of waiting until I'm back at my desk, I wanted to make it possible to simply tell my computer what to do.

That became the idea behind Remote Control, a way to control your computer remotely using natural language.

What I Built

Remote Control lets me open a web app from anywhere, give my computer an instruction, and have AI carry it out for me.

It uses two AI models working together. The first model takes my request and plans what needs to happen. The second model looks at screenshots of the computer and determines the exact mouse and keyboard actions needed to execute each step.

A filesystem watcher also keeps track of the computer's filesystem structure and metadata, giving the planning model additional context about the environment it is working in.

The desktop worker itself stays intentionally simple. It captures screenshots, executes mouse and keyboard inputs, and communicates with the server. The AI does the reasoning.

How I Built It

The system is split into three main pieces:

• Web App: The interface where I create and monitor tasks.
• Web Server: Coordinates tasks, AI calls, and communication with the connected computer.
• Desktop Worker: Runs on my Windows computer and executes the AI's mouse and keyboard actions.

The web app communicates with the server, while the desktop worker maintains a connection to the server and continuously sends the computer's current state as tasks progress.

The AI operates entirely through the computer's visual interface. It doesn't need a custom integration for every application. If a human can see it and interact with it using a mouse and keyboard, Remote Control can attempt to control it.

Challenges

The biggest challenge was getting an AI model to reliably translate a high-level instruction into precise GUI interactions.

It's one thing to ask an AI to explain how to send an email. It's another thing to have it look at a screenshot, identify the correct button, determine where to click, type the right information, and continue from the resulting screen.

I also had to design the communication between the web app, server, AI, and desktop worker so that each component knew exactly when to act and when to wait for the next piece of information.

Another challenge was working within a 24-hour hackathon. I had to focus heavily on the core idea and avoid spending the entire hackathon building infrastructure for features that weren't necessary to demonstrate the concept.

What I Learned

This project taught me a lot about building systems around multimodal AI rather than simply using an AI model as a chatbot.

I learned how important context is when asking a model to interact with a real environment. A screenshot can tell the AI what the computer looks like, but filesystem information and user preferences can give it a better understanding of what it is actually trying to accomplish.

I also learned that building an AI-powered system is less about making the model do everything and more about designing the surrounding system so the model has the right information at the right time.

Most importantly, I learned that giving AI control of a real computer introduces an entirely different class of engineering problems. When the output isn't just text but actual mouse movements and keyboard input, precision matters a lot more.

What's Next

Remote Control currently supports Windows and focuses on mouse and keyboard interaction, but the underlying concept is platform-independent.

The goal is to make the experience as simple as possible:

Built With

• go
• nextjs
• openrouter