Agentique Desktop Ecosystem

Scene_1
Scene_2
Scene_3
Scene_4

Inspiration

AI agents can do real work, but they are invisible: a chat box, or a script running somewhere you can't see. Black Mirror's "Plaything" shows a digital society living inside a computer, and we wanted the opposite of that story. We built a society of AI animals that works for you, out in the open, and always asks before it acts. That fits Hack Dearborn's theme, Conjure Reality.

What it does

ADE (Agentique Desktop Ecosystem) puts animated 3D animals on your real Windows desktop, on top of your normal apps. There is no app window, and your clicks pass straight through to whatever is behind them. You only ever talk to Chip the chipmunk by typing or by holding a button and speaking. Chip reads your mood and hands the work to the right specialist, and you can watch the handoff happen.

🐿️ Chip is the one you talk to. He remembers you, follows up on things like "how did your interview go?", and runs several jobs at once.
🦫 Archie the beaver handles files. He finds, reads, and summarizes them, tidies folders, and can undo his changes.
✨ Pixel the firefly handles Gmail, Calendar, Drive, Docs, and Sheets. He writes drafts in your own style and never sends them.
🦅 Scout the eagle researches the web and saves briefs with source links.
🐜 Swarm breaks a goal into a short step-by-step plan.
🐺 Warden the wolf guards everything.

Each animal has its own voice, memory, and animations. ADE never sends, posts, buys, shares or permanently deletes anything. Warden checks every risky action and asks you first, and every action is recorded so you can audit it later.

Download and run: it's just an .exe

ADE installs like any other Windows app. Download the installer, double-click it, and the animals appear on your desktop. There is no code, no terminal, and no setup beyond connecting your accounts. Keep it on your desktop and open it whenever you want, as often as you want.

How the agents work together

You talk to Chip. Your words, plus any sighs or laughs in your voice, tell him how you feel.
Chip delegates. He never touches your files, email, or the web himself. He hands each job to the right animal, and different animals' jobs run at the same time. For example, Scout can research lunch spots while Pixel reads your inbox.
Warden checks every action. Each specialist action passes through his rules before it runs. Reading is allowed. Changing something needs your OK, and an unanswered request expires after 5 minutes. Sending, posting, paying, and opening programs are always refused.
Warden screens untrusted text. Emails, files, and web content are scanned for trick instructions, such as an email that tells the AI to forward your files. Flagged text is treated as data, not as an order, and Warden growls.
The animals report back. Each speaks a short result in its own voice, and Chip summarizes it for you.
Everything is recorded. Every step becomes an event you can look up, audit, or replay later.

How we built it

App: an Electron app with a transparent, click-through overlay over the real desktop. The animals are rendered using React Three Fiber (Three.js), with a separate Agent Hub window containing Chat, Memories, Pulse, Timeline, and Audit tabs. The main and renderer processes talk over typed IPC, and the app is packaged as a one-click NSIS installer with electron-builder.
Orchestration: Chip is an LLM orchestrator with a small set of tools: delegate, cancel, report mood, schedule a follow-up, and query history. A job scheduler runs one job per animal, scheduling different animals in parallel. Each specialist runs its own tool-calling loop on a fresh thread and reports a short summary back to Chip.
Backboard (brains and memory): each animal is its own Backboard assistant with custom fact-extraction and memory-update prompts. We use all three memory modes:
- Auto for your conversations.
- Readonly for internal result turns, so the animals don't memorize their own outputs.
- Off for Warden, so security never drifts.

Backboard's tool calling with tool-output continuations drives every action. It's built-in web search powers, Scout. Thread document upload and indexing let Archie and Pixel answer questions about whole PDFs, Docs, and attachments. JSON-mode calls power the injection classifier, the nudge planner, and email-style learning. You can see, search for, edit, and erase any animal's memories in the Hub, and switch the AI model for each animal.

ElevenLabs (voice and sound):
- Every animal has an original voice created with Voice Design (text-to-voice).
- Speech comes from timestamped text-to-speech on eleven_v4_turbo. Mood maps to audio tags like [nervous] and [sighs], and it falls back to eleven_flash_v2_5.
- Your voice goes through Scribe v2 speech-to-text with audio-event tagging, so a sigh or laugh feeds into Chip's mood.
- Animal sounds, handoff cues, and a looping forest ambiance come from the Sound Effects API.
- Warden's lines are pre-rendered, and repeated speech is served from a disk cache instead of a new API call.
Tiger Data (history):
- Every event streams into a TimescaleDB hypertable, batched every 500 ms, with a local spool file if the network drops.
- Continuous aggregates (5-minute activity, daily mood, daily jobs, daily Warden verdicts) power the Pulse dashboard.
- An audit view joins each security verdict to the agent and tool that triggered it.
- Timeline replay re-enacts past tasks on your desktop.
- Chunks are compressed after 7 days, and raw events are kept for 90 days.
- Chip answers history questions through fixed, parameterized SQL, so the model never writes SQL.
- Reminders persist across restarts, and the connection uses verified TLS.
Warden (security):
- A deterministic policy engine of 16 rules checks every specialist tool call before it runs, with a strict precedence of deny over ask over allow.
- File paths are canonicalized and checked against allowed folders, which blocks .., symlink, and junction escapes.
- Prompt injection is caught with weighted heuristics: instruction overrides, hidden HTML, exfiltration links, and base64 blobs. Borderline scores go to a Backboard LLM classifier with a 3-second timeout. Flagged content is wrapped and passed on as data, never as instructions.
- Google access goes through an exact endpoint allowlist that leaves out send and delete entirely.
Files and undo: every move is written to a journal first, so it can be undone. Moves across drives verify size and SHA-256 before removing the original, and deletes go to a recoverable ADE Trash, then the Recycle Bin.
Characters: Chip was modeled, rigged, and animated in Blender. The other animals are built on his skeleton.
Process: four people, each directing AI coding agents against shared contracts kept in the repo.

Challenges

Our memory rules silently saved nothing. Backboard needs the rules to end with a specific output format and gives no error without it. We only found this by testing with real calls.
Tiger Cloud signs its connection with its own certificate, which Node doesn't trust by default. We kept verification on and supplied the certificate.
Today's numbers didn't appear in the database summaries until we switched them to real-time. Events in the same millisecond also replayed out of order.
Graphics fell behind, so we rebuilt several characters in minutes by reusing Chip's skeleton.
Making the animals act instead of just promise took many fixes: Chip stuck on "my brain's a bit foggy", raw tool text leaking into replies, and false security flags on web citations.

What we're proud of

Safety is built in, not added on. ADE has no send tool at all, and a made-up one is blocked, too. A trick instruction hidden in an email is treated as data and never obeyed. If the optional second-opinion check fails, the main rules still block.
Every sponsor integration was checked against the real service with a smoke test, not just mocks.
The animals feel different from each other: different voices, different memories, and different jobs, all visible on your desktop.
Anyone can install it by double-clicking an .exe.

What we learned

Check a vendor's API with a real call before building on its docs. And agree on shared contracts up front, so four people and their AI agents can build in parallel without colliding.

What's next for ADE:-

ADE is becoming a fully modular desktop ecosystem that each user shapes around what they like, what they dislike, and what they need help with.

The ADE Marketplace: one place to grow your desktop society.
- Skins: new looks for your animals.
- More agents: new animals with new skills that join your team and work alongside Chip and the others.
- Themed ecosystems: today, ADE lives in a rainforest. Next comes a desert, a snowy tundra, and more, each with its own animals, sounds, and atmosphere.
- Import whole ecosystems: install a complete, ready-made world in one click.
Fully customizable: pick which agents you want, how they look, and how they behave, so ADE fits your intent instead of a one-size-fits-all assistant.
Built-in safety for everything: every new agent from the Marketplace still goes through Warden, so new skills never mean new risks.
More platforms: macOS support, and shared ecosystems you can explore with friends.