Sign | Devpost

Sign Home Page
Sign English to ASL translation and audio recording to ASL
Lessons page

Inspiration

The inspiration for SIGN came from our Introduction to Python class. A student who used sign language sat near the front of the classroom, while two interpreters stood to the side and signed the entire lecture live. Watching that experience made us recognize how much additional coordination is often required for a student to access the same lesson as everyone else.

We wanted to explore how technology could supplement that support. Our goal was to create a seamless way for sign-language users to follow classroom content. SIGN is not intended to replace professional interpreters. Instead, it explores how transcription, visual signing, and learning tools could provide another layer of access. Furthermore, we wanted a place where others can use SIGN to learn sign language.

What it does

SIGN converts spoken or typed English into a structured 3D Model that signs the speech. A user can record audio into their microphone, or enter text. The application transcribes the content, generates a signing plan, and displays an illustrative interpretation through a full-body 3D character.

The platform also includes a learning studio where users can:

Observe a demonstrated sign
Rehearse beside the 3D character
Practice using their camera
Receive feedback on measurable handshape features
Revisit phrases and lesson material

How we built it

We built the frontend with React, TypeScript, Vite, Three.js, and React Three Fiber. The 3D character uses skeletal rigging, inverse kinematics, joint constraints, handshape definitions, and structured motion sequences. Our FastAPI backend connects the interface to transcription and language-processing services. OpenAI models handle live and uploaded-audio transcription, while structured AI outputs help extract meaning and construct constrained signing plans from the movements available to the application. MediaPipe provides real-time hand landmarks for the learning studio. Rather than claiming to evaluate an entire sign, the current practice system measures specific handshape features that the tracker can observe reliably.

Challenges we ran into

Our greatest challenge was translating language into believable character movement. ASL communicates through more than handshape alone. Position, orientation, motion, timing, facial expression, and body movement can all affect meaning.

Animating a full-body character required us to work through inverse kinematics, shoulder and elbow positioning, wrist rotation, finger articulation, joint limits, and smooth transitions. Small errors could make a gesture unclear or physically unnatural. Camera tracking introduced additional challenges when hands moved quickly, overlapped, left the frame, or appeared under poor lighting.

We also had to determine where generative AI was useful and where deterministic controls were necessary. Unrestricted generated movement would be unreliable, so we constrained AI output to structured plans and passed those plans through our motion and validation systems.

Accomplishments that we're proud of

We created a 3D character capable of translating spoken or typed sentences into signing animations. We also developed an interactive lesson system where users can watch signs, practice them with a camera, and receive feedback based on their hand movements.

We are especially proud that SIGN combines speech recognition, AI, 3D animation, and computer vision in one accessible learning experience. The project supports both real-time communication and guided ASL education.

What we learned

We learned that accessibility technology must communicate its limitations honestly. A visually convincing animation is not automatically linguistically correct ASL, and a hand-tracking percentage is not a fluency score.

We also learned that generative AI works best here as one part of a larger system. Structured outputs, deterministic animation, computer vision, and explicit validation made the experience more dependable than relying on AI alone.

Most importantly, this project helped us better understand the complexity of signed languages and the importance of involving Deaf and ASL-fluent reviewers when developing technology intended for real-world use.

What's next for Sign

We want to collaborate with Deaf and ASL-fluent reviewers to validate and expand the signing library. We also want to improve the character’s hand movement, facial expressions, and transitions between signs. Future versions could include personalized lesson paths, alphabet and name-signing activities, searchable lecture transcripts, and more advanced practice assessments. Our goal is to give people another way to access spoken information while making ASL learning more interactive and approachable.