Sanket

Sanket turns Sign Language into typed text in real time, giving India's 7M+ Deaf and hard-of-hearing citizens the same direct access to chatbots, messaging and email that everyone else already has

Comment

Inspiration

There are over 7 million people in India with significant hearing or speech impairment, for whom Indian Sign Language (ISL) — not English or Hindi — is their first and most fluent language. Yet every mainstream text-input surface, from Gmail and WhatsApp to AI chatbots like Claude, ChatGPT, and Gemini, assumes the user can type or speak in a spoken language.

Voice input became infrastructure for spoken-language users years ago. Sign language never got the same treatment. Asking someone to translate their own thought into a second, less comfortable language before they can even express it isn't accessibility — it's an afterthought. That gap is what we set out to close with Sanket

What it does

Sanket is a webcam-based interpretation layer — delivered as a browser extension and a companion mobile app — that sits on top of any text field:

Watch and recognize — captures hand, pose, and facial landmarks in real time and classifies each sign on-device. Reconstruct the sentence — ISL has its own topic-comment grammar, so recognized signs are reassembled into a proper sentence, never a literal word-for-word gloss. Type it in, read it back — the finished sentence is inserted into the active field, and AI/chat replies are condensed back into a short animated ISL summary.

It's not a sign-language dictionary app — it's a universal input layer for any application.

How we built it

The pipeline runs as one continuous flow, frame to sentence:

$$\text{Capture} \rightarrow \text{Landmarks} \rightarrow \text{Classifier} \rightarrow \text{Grammar Engine} \rightarrow \text{Injection} \rightarrow \text{Reply-back}$$ Frontend & Extension: TypeScript, Chrome Extension (Manifest V3), a React-based side-panel UI, DOM injection + Clipboard API AI/ML Pipeline: MediaPipe Holistic for landmark extraction, TensorFlow.js, a temporal Transformer classifier for sign recognition, and a seq2seq grammar-reconstruction model Backend & Data: Python (Django), PostgreSQL via Django ORM, cloud storage for models and sign clips, Django REST Framework for vocabulary sync Mobile & Reply Mode: Android CameraX, AccessibilityService, extractive NLP summarization, and a 2D avatar renderer for signed replies

Webcam frames are grabbed at 20–30 fps, landmarks are extracted on-device, and the classifier turns a sequence of signs into a gloss that the grammar engine rebuilds into a grammatically correct sentence before it's typed straight into the focused field.

Challenges we ran into

ISL is far less resourced than ASL or BSL — there's comparatively little existing research, labeled data, or tooling to build on, so we had to scope carefully rather than assume prior work would carry us. Grammar, not just recognition — ISL's topic-comment structure means a literal sign-by-sign gloss reads nothing like natural sentence order. Recognizing signs was only half the problem; reconstructing them into something readable was the harder half. Open-vocabulary ISL translation is still an unsolved research problem. Rather than overreach, we deliberately scoped the MVP to a curated, domain-specific vocabulary of 50–150 signs for digital communication — an honest scoping decision, not a shortcut. Demo reliability over breadth — instead of shipping several partially-working flows, we committed to one flawless path (sign → structured sentence → editable draft → send) with a live on-screen overlay showing each recognized sign as it happens.

Accomplishments that we're proud of

What we learned

Accessibility tooling has to start from how a community actually communicates, not from retrofitting an existing interface — ISL isn't "English with hand movements," it has its own grammar that has to be modeled explicitly. On-device inference (MediaPipe + TensorFlow.js) matters as much for trust and offline usability as for latency. Scoping honestly — a bounded vocabulary instead of chasing open-domain translation — produces something that actually works, which matters more on stage (and in real use) than something broader that only half-works.

What's next for Sanket

Short-term: ship the single flawless flow end-to-end (sign → email or sign → AI prompt) with an editable draft step. Medium-term: personalization (a short calibration session per user's signing style) and stronger offline support. Long-term: a full two-way bridge — letting a hearing user type a reply and see it rendered back as sign-clip animation, closing the loop for both sides of the conversation.

Built With

Share this project:

Updates

Submission history