What it does
Aura Games is a platform featuring a collection of interactive, hand-gesture-controlled games designed to offer an immersive experience directly in the browser. The app currently features over 50 unique games utilizing various gesture mechanics. Instead of traditional controls, users use their physical movements to interact:Users can play Space Shooter by making a finger gun gesture to shoot alien invaders. Players can slice through targets in Fruit Ninja using rapid hand swipes. Advanced mechanics include Air Writing, which allows users to write in the air with their fingers while the AI recognizes the letters, and Wizard Spells, where players cast magic spells using specific hand gestures.
How I built it
I architected this platform to be a highly responsive, modern web application.Frontend Architecture: The application is built using React for the frontend, styled with Tailwind CSS for a responsive and modern look, and uses TypeScript for strict type safety. Framer Motion is used for smooth, engaging animations to give it a cinematic, console-like feel. Computer Vision & AI: I leveraged MediaPipe Hands for real-time hand detection and gesture recognition. Additionally, the Web Speech API was integrated for Voice AI, allowing users to navigate and control games using spoken commands. Backend & Infrastructure: The entire application is powered by the Base44 SDK, which handles backend services like data management, authentication, and integrations, allowing for rapid development and scalability.
Challenges I ran into
Running real-time computer vision models inside a web browser is computationally expensive. The primary challenge was optimizing MediaPipe Hands to track complex, multi-finger gestures with extremely low latency so that the gameplay felt instantly responsive.
Furthermore, architecting the state management for over 50 distinct game loops without dropping frame rates required meticulous optimization of React components and Framer Motion animations. Shifting from traditional mouse/keyboard event listeners to a continuous stream of X, Y, and Z spatial coordinates completely changed how I had to write the game physics and collision detection.
Accomplishments that I'm proud of
I am incredibly proud of successfully engineering advanced game mechanics entirely driven by computer vision in the browser.Building physics-defying experiences like Hologram Builder, where users build 3D holograms in the air using hand gestures, pushed my technical limits. Implementing robust progression mechanics, such as the health and lives system in Fruit Ninja and Space Shooter, added real arcade-level depth to the games. Integrating a cohesive Voice AI that allows users to start and navigate games completely hands-free.
What I learned
Building this platform deepened my understanding of browser-based computer vision and spatial computing. I learned how to handle high-frequency continuous data streams (camera coordinates) efficiently, optimize React re-renders for heavy graphic applications, and build a highly scalable backend architecture using Base44.
What's next for Aura Games - AI Hand-Tracking Console
The architecture is highly scalable and ready for expansion. The next phase involves:Expanded Multiplayer: Scaling the existing multiplayer options (like the ones currently in the Boxing game) to allow remote players to compete globally in real-time. Advanced Voice Control: Expanding the Voice AI to handle complex, in-game voice commands alongside hand tracking. Library Expansion: Continuing to add more physics-based mechanics to the existing library of 50+ games.
Log in or sign up for Devpost to join the conversation.