UI
Settings Panel
System Design
Data Flow

Syso - An Intelligent Real-Time Caretaker for Your Computer

Inspiration

It started with a frustrating story a teammate shared. A friend of theirs was sitting a professional certification exam, one with a pass rate above 90%, when their laptop became unresponsive mid-exam. Panic set in. They failed.

That story stuck with us, because it isn't rare. Most of us have no visibility into what our system is doing under the hood until it's too late. You don't get a warning; you get a crash. While brainstorming ideas for the Gemini Live Agent Challenge by Google Cloud, we asked: what if your computer had a caretaker? Someone or something quietly watching in the background, catching problems before they catch you.

That idea became Syso.

What We Built

Syso is an intelligent live agent that acts as a real-time health monitor and assistant for your computer. It runs quietly in the background, continuously tracking:

CPU and RAM usage
System temperature
Storage space
Overall system activity

When something unusual is detected, Syso uses the Gemini Live API to reason about the best course of action and then asks you before doing anything. All of this happens in real time, through natural voice conversation.

And it's not just reactive. You can also tell Syso to do things: "Change my theme to dark mode" and it does.

How It's Different

Tools like Copilot or Siri wait for you to ask before they act. Syso flips that model. It continuously analyses your system and proactively surfaces suggestions but only executes them after your explicit approval. That distinction matters a lot for reliability and trust.

Beyond reactivity, Syso learns. Powered by a multi-tiered RAG architecture, it adapts to your habits over time. If you always put your laptop to sleep instead of shutting it down (guilty), Syso notices, and eventually suggests an auto-shutdown schedule tailored to your usage pattern. It remembers your preferences, the apps you like always running, whether you want Do Not Disturb during meetings and builds a personalised model of how you work.

Syso is also truly multimodal: it accepts voice input, text chat, and live screenshots.

How We Built It

We designed a split architecture that separates concerns clearly:

Local Component (Python Desktop App)

A native Python app runs on the user's machine, responsible for:

Collecting system telemetry
Handling voice interaction
Executing approved system-level actions

Getting this to reliably execute system commands took significant testing. After a lot of iteration, we got it stable on both Windows and Ubuntu, making Syso cross-platform.

Backend (FastAPI + Google ADK)

The local app communicates with a FastAPI backend over WebSockets. The backend is configured to interface with Gemini models via Google Gen AI SDK. When Gemini determines an action is needed, it triggers predefined system tools, which the local app executes safely. We built guardrails throughout to ensure the agent can't take unwanted actions.

Multi-Tiered RAG Memory

This is the part we're most proud of. Syso's memory system has three layers:

Layer	Purpose
Current State Memory	Updates the system prompt in real time with live telemetry
Episodic Memory	Stores prompts, responses, and action outcomes as vectors using ChromaDB
Ephemeral / User Preferences Memory	Stores confirmed preferences, both from onboarding and learned over time

The episodic layer also acts as a filter. We implemented a scoring system combining the frequency of a user action with its success rate. Once a score crosses a threshold, the behaviour graduates into the preferences memory, meaning Syso starts treating it as a known preference, not just a pattern.

Cloud Infrastructure

Running everything locally would defeat the purpose. The entire backend is hosted on Google Cloud, taking the computational load off the user's machine.

We used Terraform for infrastructure-as-code, so if anything broke or needed updating, we could spin it back up with minimal friction. The FastAPI server runs on Compute Engine, and because our backend and the Gemini Live API share the same Google Cloud ecosystem, latency is near zero.

Security was a priority throughout. We used Google Cloud's Virtual Private Cloud (VPC) with strict firewall rules to ensure all data between the user's device and the cloud stays private.

Challenges We Faced

System command execution was harder than expected. Getting reliable, cross-platform command execution, especially across Windows and Ubuntu required a lot of trial and error before we had something stable.
Designing the memory tiers involved careful thinking about what should persist, what should decay, and how to avoid the model accumulating noisy or incorrect preferences over time.
Latency in voice interaction required us to be deliberate about what processing happened locally vs. in the cloud. Keeping the Gemini API calls within Google Cloud's own network was the key move here.

What We Learned

Agentic AI systems need human-in-the-loop checkpoints to be trustworthy. Syso only acts with approval and that constraint actually made the system feel more useful, not less.
RAG memory design is nuanced. A flat memory store isn't enough; layering memory by type and confidence makes the agent meaningfully smarter over time.
Infrastructure-as-code (Terraform) is worth the upfront investment. It saved us multiple times when things needed to be rebuilt quickly.

What's Next

Syso is an early glimpse of what personal agentic AI could look like proactive, personalised, and genuinely useful. We'd love to extend platform support, refine the preference learning model, and explore tighter integrations with the apps people use every day.