Workflow
Daily Task and Past sessions
Analytics
Learning Curve
Feature to support multiple language, Sarvam API
Sign Language Practice
Practice sessions
Comparison
Webcam using Gemini Vision
Palm Recognition
Main Interface of AI with the feature of multiple regional languages
The landing page of our website
Dashboard

Inspiration

We built EduVaani because we kept seeing our own friends lose campus job interviews for one simple reason. They could write clean code and solve complex engineering problems easily but froze the second an interviewer asked them to speak English out loud. In India English is a direct ticket to better jobs yet college classrooms make students feel ashamed of their natural accents. Traditional language apps just hand you multiple choice grammar quizzes instead of actually making you speak.

At the same time deaf and hard of hearing students have almost zero modern learning tools built for them. The few sign language apps out there use American Sign Language which is completely useless for Indian students who communicate through Indian Sign Language.

English in India is about career mobility and dignity. We wanted to build a patient and practical companion where students can practice spoken English in realistic Indian situations understand explanations in their mother tongue and learn authentic Indian Sign Language directly through their laptop webcam.

What it does

EduVaani is a voice text and gesture practice companion built for Indian learners:

Interactive Spoken Roleplays: Instead of textbook drills students speak with an AI voice tutor named Vaani in practical Indian scenarios. You can practice clearing tech job interviews, bargaining with shopkeepers in an Indian bazaar, discussing symptoms with a doctor, talking with school teachers, asking airport gate questions, and lively cricket match talk.

8 Indian Regional Languages: Students speak in English and if they ever struggle EduVaani translates and explains concepts in their mother tongue across Hindi Marathi Gujarati Tamil Telugu Kannada Bengali and Indian English.

Real Time Grammar Coaching: After each sentence the tutor analyzes phrasing estimates a fluency score and gives gentle tips on how to speak more naturally with simple Hindi explanations.

Live Indian Sign Language Studio: Using a standard webcam students practice signing live. The app tracks 21 hand joints per hand in real time and checks gestures against 324 authentic Indian signs and all 26 alphabet fingerspellings aligned with ISLRTC standards.

Text to Sign Animation: Learners can type or speak any English sentence and watch animated hand sign sequences structured in authentic Indian Sign Language Subject Object Verb order.

Progress Analytics: Tracks daily speaking minutes against a 15 minute daily goal displays fluency score trends over time and provides interactive 3D vocabulary flashcards with audio pronunciation.

How we built it

We designed EduVaani around a privacy first edge architecture so heavy computer vision and speech models run smoothly on standard laptops without expensive cloud servers:

Frontend Architecture:

React 19: Handles responsive component rendering and UI updates while background threads process vision math.
TypeScript: Enforces strict end to end type safety across multimodal message schemas sign animation vectors and coaching payloads.
Vite 8: Powers lightning fast development builds and compiles production bundles in under 600 milliseconds.
Tailwind CSS v4: Delivers our custom design system using warm Indian cultural palette tokens like Terracotta Turmeric Peacock Teal Warm Sand and Warm Ivory with zero runtime CSS overhead.
React Router 7: Manages client side routing across the landing page voice tutor dashboard sign studio and analytics tabs with clean history state.
Framer Motion: Powers physics based spring animations interactive card flips and smooth floating control docks.
Radix UI Primitives: Accessible unstyled primitives for dialogs dropdowns progress bars and tooltips with full keyboard navigation and ARIA support.
Canvas Confetti: Renders lightweight celebration particles when learners hit speaking streaks without slowing down the DOM.

Computer Vision and Hand Tracking Stack:

MediaPipe Tasks Vision (@mediapipe/tasks-vision): Compiles Google hand landmark detection into WebAssembly with GPU delegates. It tracks 21 three dimensional keypoints on each hand at 30 frames per second directly inside the browser. Camera frames stay strictly inside local memory and never leave the student device ensuring complete privacy and zero cloud vision costs.
Custom Geometric Classifier (signLanguage.ts): Since MediaPipe only returns raw coordinates we wrote custom vector math calculating Euclidean distances joint angles and palm directions to detect two handed Indian Sign Language gestures and fingerspelling.
Animation Keyframe Sequencer (signGeneration.ts): Maps English words to 324 custom keyframe datasets and finger coordinate transitions in Subject Object Verb order.

AI Brain and Voice Pipeline:

Google Generative AI SDK (@google/generative-ai): Connects to Gemini models with structured system instructions for Indian roleplay personas and pedagogical stages progressing from warm up to teaching practice and review.
Sarvam AI API (Bulbul v3 and Mayura): Streams lifelike Indian accented speech synthesis using speaker voices like Kavya and Aditya and handles regional translations.
Web Speech API: Built in as a zero cost zero latency fallback for speech recognition and voice synthesis supporting Indian English (en-IN) and Hindi (hi-IN) so the tutor works even without API keys or on weak connections.

Code Quality:

Oxlint: Ultra fast Rust based linter maintaining strict code standards across the project.

Challenges we ran into

Running Computer Vision on Budget Laptops: Calculating 42 joint landmarks across two hands in real time caused frame drops on low cost hardware. We switched model assets to lightweight float16 formats optimized vector distance checks and decoupled vision updates from React state renders to keep a solid 30 frames per second.

Eliminating Audio Chaining Delay: Running speech recognition AI generation translation and neural speech synthesis in sequence initially created an awkward multi second pause between turns. We fixed this by streaming audio running live interim speech transcription and tuning prompt tokens so the tutor replies almost instantly.

Indian Sign Language Linguistic Authenticity: Most open source tools only support American Sign Language which relies on single hand spelling and English grammar. Indian Sign Language relies heavily on two handed midline symmetry prayer handshapes like Namaste and Subject Object Verb syntax. We hand built gesture vector models and animation keyframes for 324 signs from scratch.

Viewport Containment and Chat Auto Scrolling: Building a chat interface with live audio waveform bars expanding coaching cards and camera feeds inside fixed mobile and desktop viewports required custom scroll anchoring so new messages automatically glide into view without scrolling the entire browser window.

Accomplishments that we're proud of

Running 3D hand landmark detection completely inside browser WebAssembly with zero external server dependency.
Curating an authentic library of 324 Indian Sign Language animations and all 26 alphabets.
Creating an inclusive learning environment where Indian students can practice English freely without feeling self conscious about their accent or native language.
Delivering a clean production build that compiles in under 600 milliseconds with zero runtime errors.

What we learned

Designing for accessibility from day one forces you to build cleaner lighter and more reliable frontend architecture.
Spoken confidence comes from low pressure realistic conversations where you feel safe making mistakes rather than memorizing grammar rules.

What's next for EduVaani

Continuous Indian Sign Language sentence recognition with head movement and facial expression tracking.
An offline practice mode for rural community centers with unstable internet connectivity.
Vocational scenario modules covering railway bookings banking interviews and government paperwork.