plARes

A living WebAR AI that evolves with you. Experience zero-latency battles in your living room where your robot sees, hears, and adapts using the power of Gemini Live API.

Comment

Inspiration: Beyond the Screen, Into Your Reality

"Stop typing, and start interacting." When we heard the call to move beyond the text box, we envisioned a future where AI isn't confined to a chat window. What if your AI could live in your room, understand your voice with zero latency, recognize your environment, and battle alongside you?

This inspired plARes, an app-less WebAR entertainment system. We wanted to build a "Living AI Partner" that fuses the thrill of high-speed action games with the deep, emotional connection of a multimodal AI agent, bringing the joy of DIY and crafting into the digital arena.

What it does

plARes is a next-generation AR action game where your AI-driven robot character lives, evolves, and battles in the real world.

  • Zero-Latency Combat (Priority 1 Reflexes): Players issue voice commands (e.g., "Dodge left!") via the local Web Speech API for instantaneous, lag-free evasion.
  • Multimodal AI Strategy (Priority 2 Brain): The Gemini Live API acts as the "tactical brain," analyzing the battle and issuing macro-commands via the Agent Development Kit (ADK).
  • Native Audio "Spirit" Judging: When casting special moves, the AI evaluates your 16kHz raw PCM voice stream, scoring your "Accuracy," "Speed," and emotional "Passion" to determine attack power.
  • Reality-Fusion Crafting: Spectators can snap a photo of a real-world object (e.g., a fried spring roll) and type "Legendary Laser Sword." Using Gemini 3 Flash Image (Nano Banana 2), the system fuses the texture and prompt into a 3D weapon and drops it into the AR arena within seconds.
  • Dynamic Persona & Server-Driven UI (SDUI): The robot's personality evolves based on battle history. Post-match, the AI generates localized "Tactical Options" and even delivers victory speeches in the defeated player's native language.

How we built it

We architected a highly scalable, asynchronous hybrid system to balance real-time action with complex cloud AI processing:

  • Frontend (The Body & Senses): Built with React, Three.js, and WebXR. It handles spatial awareness (Hit Test & Depth Sensing) and zero-latency local voice input.
  • Backend AI (The Brain): Deployed statelessly on Google Cloud Run. We utilize the ADK (Agent Development Kit) to orchestrate multiple specialized agents (Tactician, Voice Judge, Commentator).
  • Latency Hiding (The 3-Second Rule): To mask the 1-2 second AI inference delay, we implemented a "Motion-First Design." When a special move is called, the frontend instantly plays a 3-second charge animation while the Gemini Live API processes the voice stream in the background, ensuring a seamless UX.
  • Data & Memory: We use Firestore for long-term memory persistence and Vertex AI Context Caching to reduce input token costs by up to 70% while maintaining vast battle logs.

Challenges we ran into

Integrating a cloud-based LLM into a real-time, low-latency action game was incredibly challenging. Initially, waiting for the AI to process commands caused the game to stutter and break immersion. We solved this by designing the 3-Tier Priority Control (FSM): decoupling local reflexes (Priority 1) from cloud strategy (Priority 2) and base pathfinding (Priority 3). Furthermore, implementing strict protocol separation—using WebRTC Data Channels for coordinates (10-30ms lag) and WebSocket for JSON events—was crucial for maintaining perfect AR sync.

Accomplishments that we're proud of

We successfully proved that you don't have to sacrifice real-time gameplay to utilize powerful generative AI. By combining Latency Hiding techniques with the Gemini Multimodal Live API, we created an experience that feels truly alive. We are also incredibly proud of our CC0 Asset Pipeline and Server-Driven UI, which allowed us to build a commercial-grade, multilingual architecture within the hackathon timeframe.

What's next for plARes

We plan to expand the "Walk Mode", allowing the AI to use Bidi-streaming to interact with the real-world environment outside of battles (e.g., commenting on a nice wooden bench). We also aim to fully implement procedural victory songs using Lyria and batch generation of cinematic highlights using Veo.

Built With

Share this project:

Updates

Submission history