Geist

• 

  Boo posing for a screenshot v1

• 

  Boo posing for a screenshot v2

Inspiration

Geist started as a question: what if an AI assistant did not live in a chat box?

Most assistants wait for commands. Geist is an experiment in making an AI feel present inside the computer itself: a small embodied screen pet that can see what is happening, think about it, move toward things it notices, and talk with the user in realtime. The goal was not just to make a useful agent, but to explore a different interface for AI: something spatial, reactive, emotional, and alive on the desktop.

What it does

Geist runs as a fullscreen transparent Electron app. A little ghost floats over the desktop, watches the screen, listens, speaks, forms inner thoughts, and wanders toward visual points of interest. It has a visible “mind” panel where perception, planning, thoughts, and state changes surface as they happen.

Instead of only answering questions, Geist behaves like an embodied presence. It notices text, windows, cursor activity, camera context, and conversational pauses. It can decide to look at something, drift over to inspect it, think privately, or speak unprompted when something seems interesting.

How we built it

The app combines several loops:

• An Electron transparent overlay for the actual desktop companion.
• A custom animated renderer for Geist’s body, face, gaze, mood, idle motion, and movement.
• OpenAI Realtime voice for fast conversational speech.
• Vision/perception loops that capture screen and camera context.
• A lightweight planner that turns visual interest into waypoints and inner thoughts.
• A brain/thought panel that exposes what Geist is noticing and planning.
• Local state and memory controls so the character can maintain continuity without becoming overloaded.

A lot of the work was about embodiment: making gaze direction match attention, making motion waypoint-driven, making speech reflect what Geist actually sees, and making the character feel like it inhabits the screen rather than commenting from outside it.

What we learned

The hard part was not simply connecting APIs. The hard part was coherence.

A believable embodied AI needs many small systems to agree with each other: voice, vision, movement, gaze, memory, personality, timing, and interruption behavior. If the model says it sees something but the eyes look elsewhere, the illusion breaks. If it speaks with no goal, it feels random. If it moves without visible motivation, it feels decorative instead of alive.

We learned that embodiment is a product problem as much as a technical one. The character needs goals, constraints, rhythm, and a grounded relationship to the environment.

Challenges

The hardest challenges were:

• Getting screen vision to show up in dialogue, not just internal thoughts.
• Making Geist move freely without drifting off-screen.
• Keeping the persona specific without becoming gimmicky.
• Avoiding prewritten sequences that felt fake compared to realtime generation.
• Balancing unprompted behavior so Geist feels alive but not annoying.
• Making the “mind” panel expose useful internal state without turning into noise.

Collaboration

This was built quickly with AI pair-programming. Claude was especially helpful for greenfield exploration and large feature drafts. Codex helped stabilize the project, fix regressions, tighten behavior, debug broken movement and perception loops, improve documentation, and prepare the final submission assets.

That collaboration itself became part of the project: an AI-assisted build of an AI companion that is trying to become more embodied, coherent, and present.

Built With

• ai-agent
• companion
• desktop
• electron
• gpt-5
• javascript
• multimodal
• openai
• realtime-api
• screen-awareness
• speech-to-speech
• three-js
• vision
• voice-ai