SignDojo

Inspiration'

Learning American Sign Language can be difficult without someone available to demonstrate signs and provide immediate feedback. Most tools rely entirely on videos or static images, which can make practice feel passive. We wanted to create something more interactive: a physical practice partner that combines computer vision, gamification, and robotics. That idea became SignDojo.

What it does

SignDojo is an iOS learning app connected to a 3D-printed robotic hand. The app introduces users to ASL through guided lessons, challenges, rewards, and progress tracking.

Using the phone’s camera, SignDojo identifies hand landmarks and processes the user’s hand position with a pretrained CNN. The app can then compare the detected gesture with the expected sign.

The app also sends commands over Bluetooth Low Energy to an ESP32-controlled robotic hand. Seven MG995R servo motors move the fingers, wrist, and forearm so the robot can achieve different hand shapes.

How we built it

We divided SignDojo into three connected systems:

The iOS app was developed in Swift, with Supabase handling backend data and user progress.
The computer-vision pipeline uses OpenCV and a Hand Landmarker model to locate important points on the user’s hand. NumPy and SciPy preprocess the landmark data before it is passed into a pretrained CNN stored in ONNX format.
The robotic hand was designed in Onshape and fabricated with PLA using 3D printing. An ESP32 DevKit V1 runs Arduino C++ firmware and controls seven MG995R servo motors. Nylon fishing line acts as artificial tendons, while elastic cord helps return the fingers to their resting positions. The system is powered by a 7.4-volt LiPo battery.

Bluetooth libraries including BLEDevice, BLEServer, BLEUtils, and BLE2902 connect the app to the ESP32. We used ESP32Servo to translate received commands into coordinated hand movements.

Challenges we ran into

One of our biggest challenges was coordinating seven servos without creating inconsistent or unnatural finger movements. Each tendon required a different amount of tension, so we had to repeatedly adjust the fishing line, elastic cord, and servo angles.

On the software side, we worked through differences between raw camera input, hand-landmark coordinates, and the input format expected by the CNN. Connecting the Swift app, machine-learning pipeline, Bluetooth communication, and physical hand into one reliable experience also required extensive integration testing.

Accomplishments that we're proud of

We are especially proud that SignDojo connects a polished mobile experience to a functional physical prototype. A lesson selected in the app can travel through Bluetooth to the ESP32 and become a coordinated gesture on the robotic hand.

We also created a complete learning experience rather than an isolated hardware demonstration. SignDojo includes lessons, computer-vision recognition, progress tracking, rewards, and a robotic instructor, all built around making ASL practice more engaging.

Finally, we designed and assembled the robotic mechanism ourselves using accessible materials and off-the-shelf components, keeping the estimated hardware cost under $90.

What we learned

We gained experience with CAD, 3D printing, ESP32 development, Bluetooth Low Energy, Swift, computer vision, and machine-learning preprocessing.

Most importantly, we learned that ASL is much more than a collection of static hand shapes. Movement, orientation, facial expression, and body position all carry meaning. This shaped both the limitations of our prototype and our plans for its future.

What's next for SignDojo

Our next step is to improve the camera-based feedback system so SignDojo can give users specific, real-time guidance about finger placement and hand orientation.

We also want to support two-handed signs, moving signs, facial expressions, and more complete ASL lessons. Additional goals include adaptive difficulty, expanded progress tracking, improved servo calibration, a more portable robotic hand, and a larger library of demonstrated signs.

Ultimately, we envision SignDojo as an accessible, interactive practice partner that makes learning ASL feel less like memorization and more like training in a dojo!