Project name: Marbles
Inspiration
AI agents can do impressive work, but managing several of them still means juggling terminals, repeating instructions, and searching for results. We wanted a more natural way to work with them—something closer to directing a team. Inspired by cinematic interfaces like JARVIS, we built Marbles: a shared workspace where voice expresses intent, gestures provide context, and work stays visible.
What we built
Marbles brings Claude Code, Codex, and Cursor into one interactive canvas. Users can delegate tasks by voice, point and pinch to explore their workspace, and watch agents produce reports, spreadsheets, presentations, and code. Agents can work independently or collaborate through a shared workspace and blackboard. Finished files appear beside the agents that created them and open directly inside the canvas.
How we built it
We built the interface with React, TypeScript, Zustand, and a Canvas-based graph. MediaPipe powers hand tracking and attention detection, while a local Whisper model listens for the wake word “Marbles.”
OpenAI transcribes spoken commands, and a Claude-powered Hermes coordinator interprets requests and delegates work to installed agent CLIs. A Node.js and Express backend manages worker processes, task context, files, and live updates. ElevenLabs provides spoken replies, with OpenAI speech as a fallback.
What we learned
The hardest part of a multi-agent interface is making the work understandable. Users need to know who is working, what context they have, and where the result went.
We also learned that natural interaction needs careful boundaries. Wake detection, command listening, and spoken replies must work together without interrupting or triggering one another. Gestures need stable targets, clear feedback, and a reliable mouse fallback.
Challenges
Our biggest challenge was connecting real agent execution with voice, gestures, and live visual feedback. Different CLIs expose different capabilities and output formats, making consistent task tracking difficult.
Voice introduced latency, background noise, and the risk of spoken replies waking the system. Gesture control required balancing responsiveness with accidental selections. We also had to present many file formats clearly while keeping the main workspace calm and readable—all within a hackathon.
