Vaarta

Inspiration

Traditional language-learning apps rely on multiple-choice quizzes, flashcard drills, and repetitive translation exercises. While these can teach isolated vocabulary, they fail to prepare learners for real-world spontaneous communication-especially in Indian languages, where social context, regional registers, and cultural kinship terms (like Kaka, Tai, Dada, Aap vs Tum) dictate how people actually speak.

We wanted to create something fundamentally different: an immersive language-learning experience where you learn a language by living in it. We envisioned an AI-driven RPG where walking into a chai tapri in Rishikesh or a fish market in Kochi isn't a pre-scripted dialogue tree, but an open, living, interactive simulation that generates on the fly from a single prompt.

What it does

Vaarta is an AI-powered, 2D pixel-art RPG that turns real conversations into language lessons across six Indian languages (Hindi, Marathi, Bengali, Tamil, Gujarati, and Malayalam).

From a single prompt (e.g., "A locksmith booth in Gulmohar Lane" or a starter journey like "The Last Pahadi Bus"), Vaarta:

Authors a complete world and curriculum: Generates the setting, lore, three authentic NPCs with distinct social registers, and a 7-step CEFR-aligned Can-Do lesson ladder (from basic polite greetings to negotiation and resolving misunderstandings).
Paints living pixel-art scenes: Dynamically generates authentic regional isometric environments using Gemini image models.
Automates collision and interactivity via Computer Vision: Uses Gemini Vision to analyze its own artwork, extracting walkable paths, solid obstacles, doorways, and interactive objects without any manual level design or tilemaps.
Hosts voice and text conversations: An AI tutor evaluates spoken (via microphone) or typed utterances in real time, scoring grammar, script production, vocabulary overlap, and pragmatic intent.
Voices NPCs in native accents: Synthesizes in-character NPC replies using Sarvam AI's bulbul:v3 TTS engine for natural regional prosody.
Reinforces retention: Automatically banks words encountered during gameplay into a personal spaced-repetition system (intervals: 1 -> 3 -> 7 -> 16 -> 35 days).

How we built it

Frontend & Game Engine: Next.js 16 (App Router, Turbopack), React 19, Tailwind CSS 4, and shadcn/ui. The 2D exploration canvas, movement mechanics, collision math, and procedural minimap run on standard HTML5 Canvas and React state without bulky third-party game engines.
World & Curriculum Planning: Google Gemini 3.6 Flash structures the Game Bible, defining NPC backstories, kinship dynamics, and pedagogical milestone goals.
Generative Pixel Art: Google Gemini 3.1 Flash Image paints 16:9 regional pixel-art environments with consistent aesthetic direction.
Computer Vision Pipeline: Gemini Vision parses generated frames into geometric coordinate polygons (walkable areas vs. solid barriers) and semantic bounding boxes for interactable hotspots.
Conversational Tutor & Scoring: Gemini 3.6 Flash evaluates user speech/text against target objectives, distinguishing between unaided fluency (100% credit) and coached completions (reduced weight).
Speech Synthesis: Sarvam AI (bulbul:v3) generates audio with authentic Indian language phonetics and cadence.
Authentication & Persistence: Supabase (PostgreSQL with Row Level Security, Google OAuth, and Storage for world thumbnails and assets), paired with a fallback browser localStorage engine for guest play.

Challenges we ran into

Bridge from Generative Art to Playable Physics: Converting free-form AI-generated pixel art into a structured 2D collision map was a major challenge. Standard generative models produce pixel data with no awareness of walls or walkable floors. We engineered a vision pass using Gemini Vision that extracts polygon bounds and interactive entities directly from the image data in JSON format.
Pedagogical Evaluation vs. Immersion: When a learner makes a mistake in grammar or honorific register, a typical AI assistant simply breaks character to correct them. We tuned our tutoring prompt so NPCs react strictly in-character with natural, socially realistic clarifying questions, while the pedagogical grading happens silently under the hood.
Handling Script and Romanization Duality: Beginners often type Indian languages using phonetic English alphabets (Hinglish/Romanized), while intermediate learners use native scripts (Devanagari, Tamil, etc.). We built a deterministic Unicode script detector and vocabulary matcher to credit learners fairly regardless of input mode.
Latency Optimization: Generating a world, lesson plan, scene art, collision coordinates, and NPC voiceover involves multiple chained model calls. We implemented pre-warmed starter journeys, parallelized API execution, and background scene generation so players can begin speaking immediately.

Accomplishments that we're proud of

Zero-Manual Tilemaps: Built a fully playable, multi-screen 2D RPG engine where levels, obstacles, and portals are generated entirely from raw AI imagery without touching a level editor.
Multi-Register Indian Language Support: Successfully implemented nuanced kinship terms and honorific grammatical registers across six languages.
End-to-End Multimodal Loop: Seamless integration of image generation, vision segmentation, real-time voice recording, pedagogical intent evaluation, and native TTS speech synthesis.
Robustness & Performance: Production build compiles in under a second with optimized Next.js server components and bulletproof server-side route gating.

What we learned

How to reliably extract structured geometric coordinates from computer vision models for real-time game physics.
The depth of sociolinguistic nuance required to authentically model Indian languages in conversational AI systems.
How to design resilient hybrid architectures that seamlessly bridge cloud database persistence (Supabase) with instant, zero-friction client-side storage (localStorage).

What's next for Vaarta

Multiplayer Cooperative Conversations: Allowing two learners to explore the same town together and divide objectives (e.g., one learner asks for directions while the other orders food).
Expanded Language Catalogue: Adding more regional languages and dialects, such as Kannada, Telugu, Punjabi, and Odia.
Procedural Quest Chains: Multi-day mystery narratives where clues discovered in one neighborhood unlock deeper investigative conversations across interconnected towns.
Classroom & Educator Dashboard: Providing language teachers with analytics on student pronunciation accuracy, vocabulary retention curves, and common grammatical mistakes.