Handspell

• 

  App Logo

• 

  Screenshot of the app

Inspiration

Learning American Sign Language (ASL) usually means paying for an expensive tutor or watching static videos without knowing if you're making hand shapes correctly. We wanted a fast and private tool that watches your hands through your phone camera and gives feedback in real time.

What it does

Handspell is a native Android app that tracks your hand shapes in real time using your front camera. It guides you through ASL letters and word signs, showing landmark wireframe overlays alongside clear three-state indicators (no hand / adjust position / match). It includes all 26 alphabet letters, core word signs, daily challenges, and story-based lessons.

How we built it

We built Handspell natively using Kotlin, Jetpack Compose, CameraX, and Google MediaPipe.

CameraX passes live video frames to MediaPipe to extract 21 3D hand landmarks. To keep recognition accurate regardless of hand distance or left/right orientation, we normalize raw coordinates around the wrist origin W₀ = (x₀, y₀, z₀) and scale them by palm length:

$$p_i' = \frac{p_i - W_0}{\Vert{}p_{\text{middle_mcp}} - W_0\Vert{}}$$

Left hands are mirrored across the x-axis (x' = -x).

For classification:

• Static Letters (A–Z): A small on-device MLP and k-NN classifier handles single-frame shapes, combined with a hold-to-confirm temporal filter so the screen doesn't flicker.
• Dynamic Signs (J, Z, and Word Signs): Fingertip motion paths across frames are passed to a lightweight 1D CNN.

RevenueCat Integration: We integrated the RevenueCat Android SDK (Purchases) to power our Freemium model. Basic fingerspelling and core signs are free, while Handspell Pro ($2.00/month or $10.00/year) unlocks extra word signs, unlimited speed challenges, and story lessons. Both plans map to a single handspell_pro entitlement. RevenueCat handles purchases, renewals, and entitlement checks on-device. We also added a local 3-day demo mode so testers can preview Pro features without hitting live store APIs.

Challenges we ran into

• Look-alike hand shapes: Letters like A, E, M, N, S, and T all look like fists. We had to focus on small thumb-over-finger joint offsets to tell them apart.
• Face and body overlap: Signs near the face or chest (like "mom" or "sad") caused tracking issues when hand points overlapped with facial landmarks.
• Performance: Running vision models and normalization on every camera frame choked the main thread at first, so we moved landmark extraction to background coroutines.

Accomplishments that we're proud of

• Getting real-time gesture recognition running smoothly on-device at 60 FPS without battery drain or screen stutter.
• Keeping 100% of video processing private in temporary memory without sending a single frame to a server.
• Designing a clean UI with nested corner geometry where inner card radii scale naturally: r_inner = max(0, r_outer - inset).

What we learned

We learned that high model accuracy on paper doesn't guarantee a good app experience. Raw predictions were originally too twitchy, so adding a brief temporal buffer sacrificed a fraction of a second in raw speed but made feedback feel smooth and forgiving. We also learned how to handle continuous video streams in CameraX without overloading Android's main thread.

What's next for Handspell

We're working on publishing Handspell to the Google Play Store. After that, we plan to expand our sign vocabulary, improve face-overlap tracking, and partner with local educators to bring Handspell into introductory ASL classes.

Built With

• ai
• android
• camerax
• image-recognition
• javascript
• jetpack-compose
• kotlin
• machine-learning
• mediapipe
• python