Human children passively learn language by hearing 1000s of words spoken around them before eventually speaking their own first one. Blind-deaf children cannot as easily perceive their environment and instead need to be actively taught Braille or sign language to communicate. As a consequence, they often develop their cognitive and social skills much more slowly, and in the long run suffer from isolation or struggles with integration into adult work life.

What it does

Sertio is a language learning assistance system, which enables blind-deaf children to more passively learn words as they explore their environment- without supervision of a parent or teaching specialist. Sertio is a learning kit consisting of a small discrete wearable camera, a braille-SmartWatch and an App that allows parents to track their child's learning. The deaf-blind child wears the smartwatch and the camera (mounted on a pair of glasses), and as it explores its surroundings and feels different objects. Sertio uses Object Detection to identify the object and display the braille word on the smart watch. This way, the child can feel the word on the watch over and over as he plays with the same objects again, and over time and can learn the new words. The app gathers a list of the words that a child got displayed on his phone, so that the parents know what the child has learned.

This way, children that are normally highly dependant on a teacher can gain language skills through their own exploration. This can give them an essential head-start and confidence in pursuing their further education and reaching their full academic capacity.

How we built it

We used Microsoft's Cognitive Services for Object Detection. We also used a public project for Hand detection in order to robustly detect the relevant object in the child's hand. We used Swift for the UI and App development. For the communication between our AI system on the cloud on the App, we used a Flask Web App.

Challenges we ran into

Accomplishments that we're proud of

After a long night of brain storming and researching we're proud to have come up with a unique idea, that has not been tried ever before. In addition, we're very thrilled to contribute towards the well-being and empowerment of an underserved group of people, who we think can really benefit a lot from this system.

What we learned

What's next for sertio

Language learning is a complex process and many linguists have come up with smart theories how to best approach it. In order for our system to actually work, this has to be incorporated. Examples of this could include: Using video data instead image data for verb learning (using LSTMs or similar). Learning hierarchy of words by displaying multiple words after each other: tool, hammer, wooden hammer.

Built With

  • azure
  • azurecognitiveservices
  • azurefunctions
  • azurerestapi
  • braillewatch
  • python
  • swiftui
  • tragbarekamerabrille
Share this project: