Hypr Realm

Hypr Realm turns Hyprland into an AI-native compositor: agents operate real apps in separate, visible desktops working in parallel while you keep using your computer uninterrupted. No VM needed.

Comment 2

Hypr Realm

Inspiration

Why can't an AI agent use my computer while I keep working, independently, at the same time?

That question came out of watching GPT-5.6 ("Sol") drive my desktop through a migration I'd been avoiding for months. It finished in about four hours, which was impressive, but while it worked, I was locked out of my own machine. One pointer, one keyboard, one active user: that's not a limitation of the agent, it's a limitation of the compositor.

After sketching the idea out with ChatGPT, I realized the fix didn't need to be a whole virtual machine per agent. The compositor itself could offer separate, visible, agent-facing workspaces alongside the one I was using. That became Hypr Realm.

What It Does

Hypr Realm is an experimental fork of Hyprland that lets humans and AI agents use the same computer at the same time.

I keep using my normal desktop. An agent, meanwhile, can spin up a Realm, a nested, compositor-managed Hyprland surface with its own Wayland display, virtual pointer and keyboard, process lifecycle, capture stream, and workspace placement. Concretely: an agent can open Firefox in a realm, search for flights, click through results, and fill out a form, while I keep editing a document in my own workspace the entire time, never losing focus or control.

Several agents can run separate realms in parallel. Every realm exposes Take Over, Pause, and Stop controls, so I can reclaim any of them instantly.

How I Built It

Built over two intensive days with Codex, adding roughly 14,000 lines of code to the Hyprland codebase.

ChatGPT turned the initial idea into a milestone plan; Codex then inspected the Hyprland codebase and expanded that into 13 implementation milestones, each one keeping the compositor buildable and testable. Roughly, the work moved through four phases:

  1. Prove it's viable. Get Hyprland running safely as a nested compositor, with native realm lifecycle and process supervision, exposed through hyprctl.
  2. Build isolation. Tie each realm to its own visible host window, give humans and agents exclusive input ownership, and open a private, bounded realm control socket.
  3. Make input and capture trustworthy. Add per-realm virtual pointer/keyboard control, capture only the realm's output (never the host desktop), and enforce computer-use permissions.
  4. Make it usable. Build a standalone MCP adapter, launch apps directly inside realms, add operator controls and a distinct agent cursor, and let agents create, operate, and clean up multiple realms dynamically.

The final architecture keeps MCP outside the compositor entirely: Codex talks to a standalone MCP adapter, which sends bounded commands through Hyprland's private realm control socket. Hyprland itself still owns lifecycle, workspace placement, input ownership, capture permissions, and the visible operator controls.

NixOS made the whole process survivable. Rather than installing an experimental compositor over my working desktop, I packaged Hypr Realm as a reproducible Nix package with a locked dependency graph, keeping Hyprland and libraries like Aquamarine and Hyprutils on compatible versions. Nix dev shells made builds repeatable; rollback-safe system generations let me test the fork in a real session and fall back to a known-good desktop the moment something broke, which it did. At one point, mismatched dependencies caused login instability and repeated desktop-portal reconnects. Nix made that reproducible, let me package the compositor and its matching portal together, and gave me a clean way back.

Challenges I Ran Into

Not breaking my own desktop. Every realm process, socket, log, and child application needed supervision and cleanup, all while I was actively using the machine the fork ran on.

Nested compositors are subtle. XWayland caused teardown instability, Unix socket paths hit strict length limits, and mismatched Nix dependencies could produce a compositor that built cleanly but misbehaved after login.

"Sent" isn't "processed." Early versions reported clicks and keystrokes as "queued," so the agent would act again before the nested compositor had actually handled the first input. I replaced this with synchronous input acknowledgements, plus a short settling period after shortcuts so browser focus changes could complete before the next action fired.

Capture had to be exact. The agent needed native realm dimensions, visual-change detection, and a visible pointer, but could never see the host monitor. Frames are pulled directly from the nested compositor and moved through shared memory, not screen-scraped from the host.

The last mile was UX, not architecture. Duplicate cursors, an invisible agent pointer, clicks landing before focus had actually changed, browser-profile quirks, offscreen rendering, desktop-portal processes reconnecting after every compositor change: none of these show up until you're testing live, and fixing them is what turned this from a proof of concept into something usable.

Accomplishments I'm Proud Of

Hypr Realm works end-to-end, not as a mockup, in two days. An agent can:

  • Create a temporary realm on a requested workspace
  • Launch a browser or other Wayland application inside it
  • Observe the realm without ever seeing the host desktop
  • Move a distinct, visible agent cursor
  • Click, scroll, type, and use keyboard shortcuts
  • Run multiple realms independently
  • Pause, resume, inspect, and clean up its own work

Meanwhile I never lose my own desktop. I can take over, pause, or stop any realm directly from its window at any time.

What I'm most proud of: this isn't a layer bolted on top of a desktop. It's implemented inside a real compositor, packaged for NixOS, exposed through both hyprctl and MCP, covered by automated tests, and validated on real browser tasks.

What I Learned

The agentic era is still early, and some of its hardest problems don't get solved by adding another app on top of the existing desktop. They require rethinking layers as fundamental as the compositor. Today's desktops assume one human actor; the next generation needs to coordinate humans, assistants, and autonomous agents at the same time.

Virtual machines still matter when you need strong isolation. But they shouldn't be the default answer to computer use. For a lot of local tasks, a compositor-managed work surface is more direct, more efficient, and just feels better.

And architecture is only half the job. Cursor visibility, focus timing, input acknowledgements, cleanup, human override: these small details are what separate an impressive demo from something people actually trust.

What's Next

This is a first step. The next one is stronger OS-level isolation: enforceable filesystem and network policy, clipboard mediation, secret delivery, process namespaces, resource limits, and better per-application profile separation. Realm isolation today is input and display isolation, not yet a security boundary, and closing that gap is the priority.

I also want a real interface for creating, arranging, monitoring, and assigning tasks across multiple realms, and to work out what happens when two agents contend for the same app or workspace, which isn't handled yet.

The experience I'm most excited about is an AI mentor. Today, an AI tutor can explain something or take the screen over, but it can't work alongside a student. With realms and multi-actor input, a mentor could sit inside the computer with you: talk through a problem, point at something on screen, demonstrate a hard step with the mouse, then hand control back so you can keep going.

Hypr Realm is a glimpse of that future: a desktop where humans and intelligent agents don't compete for control, but work side by side.

Built With

Share this project:

Updates