Swaya-Vibe

Inspiration

Chess is traditionally a visual and silent game, which creates a massive barrier for the 300 million visually impaired people worldwide. Currently, playing requires expensive tactile boards costing over $500 or the help of sighted assistants. For beginners, the game can also feel lonely and intimidating. We were inspired to solve both problems simultaneously by creating a "Curb Cut Effect"—solving a hard accessibility challenge to create a better learning tool for everyone.

What it does

Swaya-Vibe transforms any standard smartphone or laptop into an intelligent, talking chess coach. By streaming a live camera feed directly to the model, it:

• Sees the physical board state (FEN) instantly, handling shadows and angles via spatial reasoning.
• Reasons about game strategy, identifying threats and opportunities.
• Speaks naturally to the player, acting not just as a calculator, but as a supportive friend who helps you "Suno. Socho. Jeeto." (Listen. Think. Win.).

How we built it

The project was built entirely in Google AI Studio using Gemini 3 Pro as the core intelligence.

• Frontend: We used React (v18+) and TypeScript to build the user interface.
• AI Processing: We leveraged Gemini 3 Pro's Native Multimodality to replace complex computer vision pipelines.
• Voice & Vision: We implemented the Web Speech API for text-to-speech capabilities and the MediaDevices API to capture live camera feeds.
• Logic: Custom hooks (useSpeech) and services (geminiService.ts) manage the flow between the user's move, the camera input, and the AI's response.

Challenges we ran into

• Replacing Computer Vision: Relying on the model's spatial reasoning instead of traditional hard-coded CV pipelines to interpret the board.
• Environmental Noise: Handling real-world variables like shadows and skewed camera angles while maintaining accurate board state (FEN) detection.
• Context Management: Refining the "Move Log" and ensuring the model understands the full game context ("moveHistory") rather than just a single frame.

Accomplishments that we're proud of

• Democratizing Access: We successfully turned a solitary struggle into a shared conversation, removing the need for $500+ equipment.
• Spatial Reasoning: We proved that Gemini 3 Pro could handle complex spatial tasks (reading a chess board from a video stream) without dedicated vision models.
• The "Friend" Factor: Creating a coach that feels supportive rather than robotic, helping beginners overcome the intimidation of the game.

What we learned

• Native Multimodality is Powerful: We learned that large multimodal models can "see" and "reason" about physical objects in a way that rivals or exceeds custom-trained vision models for specific tasks.
• UI Matters: Refinements like a compact, scrolling move log are critical for making the complex data of a chess game digestible for the user.

What's next for Swaya-Vibe

We aim to continue democratizing chess by refining the "Curb Cut Effect" that makes this tool useful for both visually impaired players and sighted beginners. Future updates will likely focus on:

• Further optimizing the move log and context sharing for longer games.
• Enhancing the "supportive friend" persona to provide even more personalized coaching strategies.
• Adding features like "Undo Move" and "Highlight AI's Best Move" to deepen the learning experience.

Built With

• gemini3
• googleaistudio
• react
• typescript
• webspeechapi