OS-Ghost: The Living Desktop

Your desktop is haunted. OS-Ghost is an autonomous, multi-modal desktop companion that 'sees' your screen and orchestrates workflows into interactive mystery game using Gemini & long-running threads.

Comment

💡 Inspiration

We spend our entire lives staring at screens, but our computers feel dead—static tools waiting for input. We wanted to change that. Inspired by the "Ghost in the Machine" trope and the new "Action Era" of AI, we asked: What if your operating system was alive? What if it could see what you see, understand your context, and interact with you not just as a chatbot, but as a companion living in the interface?

💻 What it does

OS-Ghost is a screen-aware desktop agent that overlays your entire digital environment. It's not a window you open; it's a presence that's always there.

  • It Sees Everything: Using Gemini Vision, the Ghost takes periodic snapshots of your active window to understand your context—whether you're coding, reading news, or watching a video.
  • It Reacts in Real-Time: If you're reading an article about the Enigma machine, the Ghost might surface a cryptographic puzzle related to it.
  • Hybrid Intelligence: It uses a smart router to switch between Gemini 3 Pro (for high-level reasoning and vision) and local Ollama models (for privacy-sensitive, low-latency tasks/chat).
  • Gamified Productivity: It turns your browsing history into a "detective board," generating dynamic puzzles that require you to explore the web to solve them.
  • Autonomous Actions: It features a Native Input Controller that allows the Ghost to click, scroll, and type on your behalf (with strict permission levels: Observer, Suggester, Supervised, Autonomous).

⚙️ How we built it

We built OS-Ghost as a high-performance native application using Rust and Tauri v2.

  1. The Brain (Rust + Gemini): We architected a custom SmartAiRouter in Rust that handles the decision-making loop. It manages a "Circuit Breaker" to failover gracefully between Cloud (Gemini) and Local (Ollama) inference.
  2. Multi-Agent System: We implemented a robust agent architecture including:
    • Orchestrator: Manages task delegation.
    • Critic: Reviews AI outputs for quality and safety.
    • Planner: Decomposes complex goals into steps.
    • Guardrail: Enforces strict safety policies.
  3. The Eyes (Vision): We built a custom screen capture pipeline using the screenshots crate that feeds optimized image buffers directly to Gemini 3 Pro for analysis.
  4. The Nervous System (Chrome Bridge): A custom Chrome Extension (MV3) communicates via Native Messaging to the Rust backend, allowing the Ghost to know exactly which tab is open and link visual context with URL possibilities.
  5. The Face (React): The UI is a transparent, click-through window built with React and CSS animations, designed to look like a "glass" layer floating above your OS.

🚧 Challenges we ran into

  • Context Window Engineering: Feeding continuous screen context to an LLM is expensive. We had to implement a "Significant Change Detection" algorithm to only query Gemini when the visual state changed meaningfully.
  • Native Messaging Hell: Getting a sandboxed Chrome Extension to talk reliably to a Rust binary required deep diving into OS-specific IPC pipes and process management.
  • Privacy vs. Utility: We wanted it to be helpful but not creepy. We built a strict "Privacy Sandbox" where the user can toggle "Read-Only Mode" or block specific apps/domains from being "seen" by the ghost.

🏆 Accomplishments that we're proud of

  • Full Native Automation: Implementing cross-platform mouse/keyboard control in Rust (InputController) to give the agent real agency.
  • Adaptive Context: Using vector-based memory and Gemini's massive context window to provide relevant history without blowing up the token quota on every request.
  • Building a fully local fallback mode that runs on consumer hardware using Llama 3.2, ensuring the Ghost never "dies" even without internet.
  • Creating a UI that feels truly "part of the OS" rather than just another "Electron" app.

🔮 What's next for OS-Ghost

  • Multi-Ghost System: Different AI "personalities" specialized for coding, writing, or researching.
  • Procedural Mystery Generation: Using Gemini to write an endless, personalized mystery novel based on your actual digital footprints.

Built With

Share this project:

Updates