Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
Inspiration
Soon, people won't control one AI. They'll supervise an AI that manages other AIs. We wanted to feel what that's like before it arrives, so we built it as a game: you're a mech pilot, a voice supervisor commands your two drones, and each drone runs its own small model.
What it does
You pilot a mech with your keyboard or your hands (via webcam) and give orders by voice: "Send U2 to T3", "Take the far turret", "Recall drones".
- A supervisor agent (ElevenLabs) reads the battlefield and plans.
- For vague goals, each drone's worker model suggests its own target; the game checks every suggestion against the real state.
- AI-proposed orders need pilot approval. Voice orders are always confirmed in the Orders panel; the pilot can grant autonomy to the HUD supervisor.
- Recall always wins. Once triggered by voice, key or gesture, it immediately overrides all pending and active orders.
The HUD shows each agent's decisions, explanations and task status, right where it happens in the arena.
How we built it
TypeScript, Vite and three.js in the browser. Hand tracking runs locally with MediaPipe; no camera images leave the device. The supervisor uses ElevenLabs Conversational AI with a set of validated game tools. Drone workers run on a local model (Qwen via Ollama), with OpenAI as a fallback. We designed the HUD in Figma over real gameplay renders, as reusable components for every agent state, then built it from the same tokens.
Challenges we ran into
- Late AI decisions. A model can answer after the situation has changed. Every action carries a state token, so anything decided before a Recall is rejected as stale.
- Who's in charge. We hit real bugs where an agent's order blocked the pilot's. We rewrote the precedence rules so the pilot always wins.
- Showing AI state without clutter. We used one colour per authority: lilac means the AI is acting, gold means it's waiting for you, teal means you're in control, red means Recall.
Accomplishments that we're proud of
AI-proposed actions can't take effect without the pilot's say-so. Recall overrides everything the moment it fires. Every HUD state exists as a Figma component and maps to real game state. Over 400 automated tests run against the real simulation.
What we learned
Building multi-agent AI isn't just about connecting models. The real challenge is coordinating asynchronous decisions while keeping the human informed and in control. That shaped both our architecture and our Figma-designed interface.
What's next for StarshipMania
More drones and deeper agent hierarchies, priority-based policies, a new level (Orbital Relay, already in progress), and testing with real players to measure how much control people actually want to hand over.