Based on National Center for Education Statistic- U.S.-born adults make up two-thirds of adults with low levels of English literacy skills in the United States. Non-U.S.-born adults comprise 34 percent of the population with low literacy skills Colorful and simple for kids and non-english speakers.

What it does

Through ESLAssistant, we wish to tackle the difficulties by non-english speakers to learn the language by providing textual description and voice recognition through image captioning. link Alt text

How we built it

We used python with libraries such as tensorflow, and numpy to built a machine learning model to predict the caption. Specifically, we used the flicker8 dataset compiled by University of Urbana Champaign to train our model. After training the model we used tensorflow.js, javascript, HTML, and CSS to convert the model into a UI application. link Alt text

Challenges we ran into

For training, we had to use local CPU which took a lot of time. Tensorflow.js was released in 2018 so it is a relatively new library which is difficult to learn. link Alt text

Accomplishments that we're proud of

Understanding basics of Convolutional Nets and Recurrent Nets to create sensible sentences based on the picture. Integrating the model with front end so that it can give live predictions, and also set up the basis to build online training like the one used by Amazon. link Alt text

What we learned

Integration of Machine learning in the front end is difficult through tensorflow.js because it uses javascript which is a flexible at useful language. This created difficulties to learn easily as flexibility meant vagueness, and vagueness in Machine Learning projects can stop programmers from completing the project to satisfaction. link Alt text

What's next for ESLAssistant (English as a Second Language Assistant)

Convert the web application into a mobile app Improve the accuracy of the app by using online training where the user are given opportunity to figure out mistakes.
link Alt text

Share this project: