Talk2MyPC

Inspiration

My work life involves a lot of talking, waiting for meetings to start in remote locations, and driving hours to meet with people. My family also has a fairly relaxed relationship with time, so I spend a surprising amount of my life waiting somewhere away from my computers while I am running research experiments or collecting data.

At the same time, I work on projects that can leave several PCs processing data, running experiments, rendering, or executing scripts for hours. These jobs usually don't need constant attention, but they do need a few minutes of management throughout the day: checking progress, changing priorities, starting the next task, or responding when something goes wrong.

So my computers can do useful work while I'm away, but I often need to be in two places at once to use them well.

That inspired Talk2MyPC. I wanted to call my computers while living my life—ask what they're doing, manage their workloads, and give them instructions through conversation without stopping what I'm doing to operate a remote desktop.

What it does

Talk2MyPC provides an agentic middle manager between your mobile device and a set of computers. This lets you manage simple tasks remotely in a controlled environment. The flaw and feature is that it doesn't let an AI run your computer; it gives it access to a preconfigured set of tools, within a harness that manages your computer within limits you configure.

How we built it

This system comes in three parts;

Client Side Apps (prototype and mobile apps):

• A client app is designed to create a simple user interface and manage the data sent to the cloud.
• A Python prototype script exists for this for testing purposes, as well as an Android-based application.

Cloud App:

• This is a simple agentic model that takes in a voice stream and chooses an end device and a computer to call the API from to respond.

Device Monitoring Systems:

• This is more complex than I am letting on; it is designed to be a priority queue for jobs that can be remotely accessed via this method of communication with the main center.

Challenges we ran into

Configuring the desktop side was honestly the hardest part. I wanted a Web GUI to monitor accuracy and debug, but that was probably the hardest part to build because of circular-logic issues, so I had to drop it for a short period. Getting Gemini to have a conversation was hard because of difficult network issues.

Accomplishments that we're proud of

Having an admittedly slow conversation with AI is nice, but what I am proud of is the system itself. I can see a product that lets me interact with my devices in a way I really like. I think this is going to be a good concept soon.

What we learned

I have used this as a mechanism to learn about how Google Cloud works and more about API's in general. I have gained some appreciation for how complex setting up some of these systems can be and the considerations that are required when using this technology.

I was surprised how easy it was to get the agentic logic to work, and the video streaming to Gemini was a nice surprise.

What's next for Talk2MyPC

• The cloud network required more steps than expected, and there were issues with the private network connection. Moving forward, I would like to get that working completely. Talking to the devices not just Gemini.
• I also want to experiment with general conversational language and how the language needs to be set up.
• I also want to add some security features to this and design it up so other people can use it and interact with it.
• I think, firstly, what I might do though is clean up the codebase a bit.

Built With

• gemini
• python