Real-Time Tamil Fingerspelling Assistant

• 

  Demo

Inspiration

Tamil fingerspelling users need fast, clear feedback while communicating. We wanted to build an assistive tool that works in both directions: recognizing Tamil fingerspelling from a webcam and converting Tamil text back into visual sign references.

## What it does

The assistant:

• Detects one or two hands from a webcam.
• Recognizes Tamil fingerspelling characters.
• Converts stable signs into Tamil Unicode text.
• Uses green/red feedback to show finger states.
• Prevents repeated characters when a sign is held.
• Uses a fist to insert spaces.
• Converts typed Tamil text into reference images for reverse communication and practice.

## How we built it

The browser captures webcam frames and sends JPEG images to a FastAPI WebSocket server. OpenCV 5 decodes and normalizes each frame. A pretrained RTMPose/RTMDet pipeline detects hands and extracts landmarks.

We use normalized landmark geometry to classify fingers as straight or bent. These finger states are mapped to Tamil fingerspelling labels using the TLFS23 dataset. A temporal stability layer commits a character only after the pose remains stable.

The application runs in Docker and is deployed on AWS ECS Express Mode with Fargate, ECR, an Application Load Balancer, and CloudWatch.

## Challenges we ran into

Hand occlusion, poor lighting, unusual camera angles, overlapping hands, and out-of-frame gestures caused detection errors. Similar one-hand and two-hand configurations were also difficult to distinguish.

We addressed these issues with confidence feedback, stable-hold detection, duplicate suppression, two-hand support, and separate evaluation of detection, finger classification, character recognition, and latency.

## Accomplishments that we're proud of

We built a working real-time, bi-directional Tamil fingerspelling assistant using OpenCV 5. The same pipeline works locally and on AWS, with a browser interface that gives users immediate feedback.

We are especially proud of making the system human-in-the-loop: users can see incorrect finger states, reposition their hands, and correct the sign before it is committed.

## What we learned

We learned that real-time fingerspelling is more than a classification problem. Temporal filtering, interaction design, visual feedback, and failure handling are critical for making recognition useful.

We also learned how to combine OpenCV 5, pretrained pose models, geometric rules, WebSockets, Docker, and AWS into one reproducible application.

## What's next for Real-Time Tamil Fingerspelling Assistant

Next, we want to improve recognition across lighting conditions, camera angles, and hand shapes. We also plan to collect more real-world evaluation data, improve two-hand gesture handling, add pronunciation or voice output, and support more Tamil signs and phrases.

The long-term goal is a reliable, accessible communication and learning tool for Tamil fingerspelling users.

Built With

• accessibility
• ai
• amazon-web-services
• cloudwatch
• docker
• ecs
• fargate
• fastapi
• fingerspelling
• javascript
• onnx
• opencv
• python
• real-time
• runtime
• tamil
• websockets