Horizon: Interactive Education at the Tip of Your Tongue

Horizon presents a fully adaptive, real-time, augmented learning environment. We revitalize online learning, with unique benefits for low-resourced instructors and students with learning disabilities.

Comment

Inspiration

As a consequence of the COVID-19 pandemic, many classrooms around the world have transitioned to a virtual setting. Although video-conference lectures have kept us safer during these times, they have limited the richness and quality of our education, especially when lesson plans lack adequate visual aids.

What it does

Horizon is built around maintaining in-class student engagement by offering context-driven image snapshots and Word Cloud summaries of the lesson topics, all at a glance. The instructor simply conducts their lesson as usual, and our model processes the continuous audiosteam using Google Cloud and NLP. After filtration and preprocessing, we identify the entities with the greatest salience scores, synthesize them as a live-list of topical images, and generate key terms to populate our Word Cloud.

How we built it

Horizon System Architecture Diagram

Challenges we ran into

This was our first time utilizing ML in a non-academic setting and we wanted to limit its usage to be practical and add value to the product. Also, implementing Google Cloud's Speech-To-Text API presented many unforeseen challenges, but it was extremely gratifying when we were finally able to demonstrate it working in real-time.

Accomplishments that we're proud of

We are proud of the relevance of this application to a problem students and teachers around the world are experiencing in their remote learning environments. When we first considered this idea for Bitcamp, we believed that GPT-3 capabilities would be necessary to implement a working product. Since it is in private-beta and not publicly available, we had to identify and implement creative alternatives. We are proud to have done so to the best of our ability.

What we learned

We learned about NLP basics, including TF-IDF, tokenization, entity extraction, and salience scores/calculations. We also strengthened our knowledge of full-stack development by working with React, a Node.js framework, in the frontend and Flask, a Python framework, in the backend.

What's next for Horizon: Interactive Education at the Tip of Your Tongue

Horizon seeks to integrate with even more powerful technologies to further broaden its service. Connecting with the Wolfram Alpha API, our model would be capable of generating formulas and graphs from standard English grammar inputs. This could enable Horizon to play a greater role in STEM lessons and discussions. Furthermore, by leveraging OpenAI’s GPT-3 language model, Horizon would be capable of highly accurate image generation for our synthesis and NLP tasks. In the future, Horizon would certainly be eager to explore both of these avenues.

Share this project:

Updates