BeyondWords

Break Communication Barriers with AI

Comment

Inspiration

Most sign-language learning tools are static video libraries — you watch a clip, try to copy it, and have no way of knowing if you actually got it right. On top of that, almost every platform teaches ASL alone, leaving Indian Sign Language (ISL) learners — a huge, underserved population — with no structured option at all. We wanted to build something that didn't just show signs, but actually checked whether you were doing them correctly, and that could turn learning into a real, two-way conversation instead of a one-way feed.

What it does

BeyondWords is an AI-led platform for learning ASL and ISL. Learners work through structured, gamified lessons (XP, streaks, levels, leaderboards) and get an AI Sign Check that reads their hand shapes through the camera in real time and gives instant feedback on whether a sign was correct. A Speech Hub adds live speech-to-text and text-to-speech so a learner can actually use what they've learned in a live conversation with a Deaf or hard-of-hearing person. An Admin Panel gives educators and institutions visibility into course content, learner progress, and quiz performance at scale.

How we built it

  • Frontend: React + Vite + Tailwind CSS
  • Backend: Java Spring Boot, split into three microservices — user-service, learning-service, and analytics-service — talking over REST APIs
  • Data: MySQL for relational data (users, courses, progress) and MongoDB for flexible content (lesson media, quiz banks, AI session logs)
  • Auth: Spring Security + JWT
  • AI: A separate Python model serves sign predictions; the analytics-service calls it per practice frame, thresholds on confidence, and logs the session for progress tracking
  • Ops: Each service is containerized with its own Dockerfile

We designed the backend as microservices from day one so the learning experience, user/auth layer, and analytics/AI layer could be built and scaled independently.

Challenges we ran into

  • Getting real signal out of a webcam frame. Sign language depends on precise, fast hand movement, so getting the gesture-recognition model to return a confident, reliable prediction — and doing it fast enough to feel real-time — took real tuning, including handling low-confidence and failed-prediction cases gracefully instead of breaking the practice flow.
  • Coordinating three microservices. Keeping user-service, learning-service, and analytics-service in sync (shared auth, cross-service calls, consistent data) added real integration overhead compared to a monolith.
  • Deciding what to claim. Early on we prototyped an adaptive recommendation engine, but keeping it honestly scoped — clearly separating what's a working AI feature (Sign Check) from what's still a seeded, rule-based suggestion list — was its own useful discipline.

Accomplishments that we're proud of

  • A working, end-to-end gesture-recognition loop: camera → AI prediction → real-time feedback → logged progress
  • A fully functional microservice backend (not just a mockup) handling auth, courses, quizzes, and analytics
  • Support for two sign languages, ASL and ISL, in one platform
  • A working real-time Speech Hub for live two-way conversation

What we learned

We learned a lot about running AI inference inside a real product loop — not just training a model, but handling its failures, latency, and confidence scores in a way that still feels smooth to a learner. We also learned the value of being precise about what's "AI-powered" versus what's simply well-designed logic — it made our product story sharper and more credible.

What's next

  • Move our current seeded recommendation list to a real quiz-accuracy-driven model
  • Expand the Sign Check model's vocabulary and accuracy with more training data
  • Add persistent accounts, global leaderboards, and cross-device sync
  • Extend beyond ASL and ISL to additional regional sign languages

Built With

  • ai
  • asl
  • computervision
  • deaf
  • edtech
  • gesture
  • havaspreingboot
  • isl
  • ml
  • mongodb
  • mysql
  • react
  • sign-language
  • speech-to-text
  • tailwind
  • text-to-speech
  • vite
Share this project:

Updates

Submission history