SignServe

• 

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for sgsl hands

Inspiration

From our everyday observations, it struck us how many simple interactions in Singapore still depend heavily on spoken communication. At hawker centres, things like asking about ingredients, checking prices, clarifying an order, or requesting takeaway can become much harder for Deaf users who rely on Singapore Sign Language (SgSL).

We wanted to create a tool that allows Deaf users to communicate more directly and independently, without always needing to type, point, or depend on another person to interpret.

What it does

Our app supports two-way communication between Deaf SgSL users and hawkers.

A Deaf user can sign to the camera, and the system recognises the intended message and converts it into English text. The message can then be spoken aloud using text-to-speech.

The hawker can reply by speaking, and the app converts the spoken response into text for the Deaf user to read.

For our prototype, we focused on common hawker-centre intents such as food allergies, vegetarian options, pork content, price, wrong orders, missing food, asking someone to repeat, and packing food.

How we built it

We built the SgSL recognition system using MediaPipe for hand and upper-body landmark detection.

For every video frame, we extract hand and body landmarks and convert them into a fixed 144-dimensional feature representation. A complete sign is recorded as a temporal sequence, resampled to a fixed length, and passed into a trained Support Vector Machine classifier.

We collected our own dataset from multiple signers and trained the classifier to recognise a small set of application-level SgSL intents.

The recognition output contains the detected intent, English meaning, confidence score, and whether the message is safety-sensitive.

We connected this recognition system to a Streamlit interface. ElevenLabs is used for text-to-speech so the recognised message can be spoken to the hawker, and speech-to-text converts the hawker's reply into readable text.

Challenges we ran into

One of our biggest challenges was recognising signs consistently across different people. Even when two people perform the same intended sign, differences in movement, speed, hand position, and body proportions can affect the model.

We also found that automatically detecting when a sign begins and ends reduced recognition accuracy. Because our model was trained using manually bounded sign sequences, inaccurate automatic boundaries could distort the sequence during temporal preprocessing. We therefore prioritised reliability and kept explicit recording boundaries for the prototype.

Another challenge was combining computer vision, machine learning, the live camera, Streamlit, and voice services without breaking components that were already working.

We also had to be careful not to assume that English phrases map directly to natural SgSL, since SgSL has its own vocabulary and grammar.

Accomplishments that we're proud of

We built an end-to-end working prototype within the hackathon timeframe.

Our system can detect hand and upper-body landmarks in real time, track an active signer, record temporal sign sequences, classify multiple communication intents, and return an English message with a confidence score.

We collected 139 usable temporal samples across two signers and achieved 89.1% accuracy on a stratified evaluation of our collected dataset.

We also connected the recognition system to a functional two-way interface, allowing recognised signs to become spoken messages and spoken replies to become readable text.

Most importantly, we created a working proof of concept for a communication problem that affects everyday independence.

What we learned

We learned that sign-language recognition is much more than detecting hand shapes. Timing, movement, body position, signer variation, and the boundaries of a sign all matter.

We also learned the importance of designing around real limitations. Adding more automation does not always make a system better; in our case, manual sign boundaries produced more reliable recognition than automatic motion-based detection.

From a software perspective, we learned how to integrate computer vision, machine learning, frontend development, and external APIs into a single workflow while keeping the different components modular.

We also gained a deeper appreciation for the importance of involving the Deaf community when building technology around SgSL, especially for validating language use and understanding real communication needs.

What's next for SgSL

Hawker-centre communication is only our starting point.

We hope to expand the system to other everyday situations where Deaf users may face communication barriers, including clinics, pharmacies, retail counters, transport hubs, government services, schools, and workplaces.

We also want to collect a much larger and more diverse SgSL dataset across more signers, signing styles, environments, and camera conditions. This would help improve recognition for users the system has never encountered before.

In the longer term, we hope to support a larger SgSL vocabulary, more natural sentence-level communication, mobile and kiosk deployment, context-specific modes for different services, and verified SgSL output for more complete two-way communication.

Our longer-term goal is not just to make ordering food easier, but to build toward a tool that helps Deaf users communicate more independently across everyday life in Singapore.

Built With

• accessibility
• assistive
• computer
• elevenlabs
• github
• language
• learning
• machine
• numpy
• pose
• real-time
• recognition
• scikit-learn
• sgsl
• sign
• singapore
• speech-to-text
• svm
• technology
• text-to-speech
• vision