Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

Inspiration

Soon, people won't control one AI. They'll supervise an AI that manages other AIs. We wanted to feel what that's like before it arrives, so we built it as a game: you're a mech pilot, a voice supervisor commands your two drones, and each drone runs its own small model.

What it does

You pilot a mech with your keyboard or your hands (via webcam) and give orders by voice: "Send U2 to T3", "Take the far turret", "Recall drones".

  • A supervisor agent (ElevenLabs) reads the battlefield and plans.
  • For vague goals, each drone's worker model suggests its own target; the game checks every suggestion against the real state.
  • AI-proposed orders need pilot approval. Voice orders are always confirmed in the Orders panel; the pilot can grant autonomy to the HUD supervisor.
  • Recall always wins. Once triggered by voice, key or gesture, it immediately overrides all pending and active orders.

The HUD shows each agent's decisions, explanations and task status, right where it happens in the arena.

How we built it

TypeScript, Vite and three.js in the browser. Hand tracking runs locally with MediaPipe; no camera images leave the device. The supervisor uses ElevenLabs Conversational AI with a set of validated game tools. Drone workers run on a local model (Qwen via Ollama), with OpenAI as a fallback. We designed the HUD in Figma over real gameplay renders, as reusable components for every agent state, then built it from the same tokens.

Challenges we ran into

  • Late AI decisions. A model can answer after the situation has changed. Every action carries a state token, so anything decided before a Recall is rejected as stale.
  • Who's in charge. We hit real bugs where an agent's order blocked the pilot's. We rewrote the precedence rules so the pilot always wins.
  • Showing AI state without clutter. We used one colour per authority: lilac means the AI is acting, gold means it's waiting for you, teal means you're in control, red means Recall.

Accomplishments that we're proud of

AI-proposed actions can't take effect without the pilot's say-so. Recall overrides everything the moment it fires. Every HUD state exists as a Figma component and maps to real game state. Over 400 automated tests run against the real simulation.

What we learned

Building multi-agent AI isn't just about connecting models. The real challenge is coordinating asynchronous decisions while keeping the human informed and in control. That shaped both our architecture and our Figma-designed interface.

What's next for StarshipMania

More drones and deeper agent hierarchies, priority-based policies, a new level (Orbital Relay, already in progress), and testing with real players to measure how much control people actually want to hand over.

What we learned

What's next for StarshipMania

Built With

Share this project:

Updates

Submission history