Learn Baby Learn

Reading practice for small kids monitored by speech to text & facial expressions

Comment

Inspiration

Because of the busy lives parents don't have enough time to help their little children read. And this can hurt children in longer run. More often when child is left unsupervised they might skip reading.

What it does

With the help of "Reaby" parents can assign reading tasks for kids and later get data which can help them analyse where the child is lagging. The data provided by "Learn Baby Learn" are :-

  1. Amount of text correctly read by the child.
  2. For which all words/sentences the child made mistake.
  3. Facial expression data to say how comfortable was the child while reading.

How I built it

I used Houndify's Speech to Text passage to capture speech of the user and convert it to text. Then compared the text with original transcript of the story or poem. With which the rating of correctly read was calculated. Used npm diff library to track the words/parts correctly read by the child. Used Tensorflow's Facial expression package to monitor child's comfort level. And integrated all on a web portal.

Challenges I ran into

  • Fetching the input text from Houndify's JSON format.
  • few deprecated features of "diff" npm package didn't allow me to use the package comfortably. The code still doesnt work on actual web page when hosted(even locally) but works when ran offline.
  • Google vision API didnt work with CocoaPods had to move to tensorflow and web based app.
  • fetching facial expression rating, storing it into array at calculate mean of expressions.

Accomplishments that I'm proud of

I have been planning for TreeHacks from past 4 months. But none of my teammates/friends got an entry. Because I am suffering from mild cold, I did not form a team and participated alone and still have made a Proof of concept worthy project. I had to change my product thrice and had to use web technology(node.js) after 3 years, struggled learning things again but did not let any of the difficulty come in between competing the Hackathon. Self believe, is my accomplishment.

What I learned

About Houndify and it's applications. Use of tensowrflow on web.

What's next for Learn Baby Learn

  • fetching facial expression rating, storing it into array at calculate mean of expressions.
  • Storing all the output data for analytics purpose.
  • Improve tech behind catching the mistakes made while reading, capturing such words for further revision.
  • With voice modulation understand where all the child stumbled upon while reading and finding irregularities in pronunciation, reading etc.
  • Portal/social network to share such videos of kids reading for other kids to inspire from and for us adults to enjoy watching little kids make mistakes while reading and improving.

Built With

Share this project:

Updates