VisionArena

An image after the game is over.
Image at the start of the game

Vision Arena

Inspiration

AI is changing software, but most games still use it as a separate layer: scripted enemies, chatbots, or background tools. I wanted to build a game where AI was part of the core experience.

Vision Arena asks: what if the player’s body, the boss, the narration, and the match memory were all connected through AI?

The result is a 2.5D boss fight where the player punches, guards, and moves in front of a webcam while an AI-powered boss learns their habits and adapts.

What it does

Vision Arena is a real-time Unity boss fight controlled through computer vision.

A webcam maps body movement into combat actions:

Extend one arm: punch
Extend both arms: heavy punch
Raise both wrists: guard
Step left or right: move

The game also uses agents:

GameMaster: coordinates the match
Enemy Agent: adapts boss strategy
Narrator Agent: names moves and gives live commentary
Coach Agent: gives player feedback
Recap Agent: creates a post-fight summary

The goal is to make the fight feel live: your body controls the action, and the AI reacts to how you play.

Tech stack

Unity + C#: combat, animation, health, HUD, arena, and keyboard fallback
Python + MediaPipe: webcam body tracking
FastAPI + WebSockets: bridge between vision, Unity, and agents
Redis: player memory, boss strategy, move history, and cooldowns
Fetch.ai / Agentverse / ASI:One: battle and commentary agents
Arize: tracing and fight evaluation
Pika: post-fight recap generation
Vapi: optional pre-fight boss phone call

How it works

MediaPipe reads pose landmarks from the webcam every frame. Punches are detected using wrist depth compared to shoulder depth, so the player must physically extend toward the camera.

Unity keeps the real-time combat local for low latency. The agents handle slower, higher-level tasks such as strategy, narration, coaching, memory, and recap generation.

The backend connects the system with FastAPI routes and WebSockets. Unity sends combat telemetry to the backend, agents respond, and Redis stores match memory so the boss can adapt across rounds.

Reliability and fallbacks

Because live demos can fail, Vision Arena includes fallback paths:

Keyboard controls if computer vision is unavailable
Mock pose events if the camera fails
Deterministic agent responses if AI services are unavailable
In-memory storage if Redis is unavailable
Mock phone-call flow if Vapi is not configured

This keeps the core game playable even if hardware or external services fail.

Challenges

The hardest part was making webcam input feel physical. I had to tune depth thresholds, handle camera mirroring, and filter unreliable landmarks so punches did not trigger accidentally.

Another challenge was separating real-time gameplay from slower AI decisions. Combat had to stay fast, while agents handled strategy, narration, memory, and evaluation.

Since this was a solo project, I had to cut scope and focus on the core demo: body-controlled combat, an adaptive boss, and visible AI systems.

What I learned

I learned that computer vision controls depend on small details like coordinate systems, confidence scores, mirroring, and depth.

I also learned that multi-agent systems work best when each agent has a clear role. Vision Arena separates match flow, boss strategy, narration, coaching, and recap generation.

Most importantly, I learned that AI demos need graceful fallbacks. The project became stronger once the core experience could run without perfect hardware or network conditions.

What’s next

Next, I would add live MYO armband EMG input, expand boss memory across sessions, improve ASI:One commentary, and polish the arena and characters for a more cinematic fight.