How many hours have you spent in learning? It is estimated that 18 million hours spent per school day in the U.S. Due to the COVID-19 epidemic, students have been taking part in lessons at home. Many teachers have struggled to adapt to online learning. How can AI help?

Scope of project

We have decided to leverage the power of to build an engaging conversational AI response engine. This engine is applied to build a virtual classroom assistant that brings better learning experience and efficiency.

Problems to solve

We identified three problems in traditional education:

  1. Lack of self-initiation to ask a question by a student;
  2. High cost in video filming for online learning; and
  3. Limited teaching efficiency in limited teacher's preparation time.


Based on the studies [1,2,3] related to the aforementioned problems, our project aims at:

  1. Improving students’ questioning skills which are important to achieve deeper learning;
  2. Providing engaging responses to motivate students' to learn; and
  3. Increasing the accessibility of learning from best teachers.


With the help of, we develop a low code solution to achieve the above objectives. Our system supports a direct and natural way to interact with technology through language. provides the followings:

  1. Personal classroom experience for students to ask questions;
  2. More personal feel realtime video response via teacher’s talking head with lip sync technology;
  3. Low code tool to help teachers to answer students' questions supported by


In the backend, the natural language understanding engine is achieved by leveraging the natural language understanding technology powered by ( to analyze user messages. Another component in the backend is the response compilation engine, which is done by integrating Google's text-to-speech service and a recent lip-sync technology published at ACM Multimedia 2020 ( Wav2Lip is written in PyTorch from Facebook. This enables us to synthesize talking-head videos. Flask server provides the entry point to access the engine. Please refer to the GitHub repo for details.

A picture showing the technology stack can be found in the image gallery.

To Do

  • Provide better analytics and insights for educators (or people in the customer service department).
  • Perform in-depth sentiment analysis from student (or customer) activity logs.
  • Add state machine support for storyboard response generation.
  • Automate the generation of the knowledgebase files associated with the app
  • Build an ecosystem for teachers to exchange teaching materials built on top of


  1. Bugg, Julie M., and Mark A. McDaniel. "Selective benefits of question self-generation and answering for remembering expository text." Journal of educational psychology 104.4 (2012): 922.
  2. Guo, Philip J., Juho Kim, and Rob Rubin. "How video production affects student engagement: An empirical study of MOOC videos." Proceedings of the first ACM conference on Learning@scale conference. 2014.

In the demo application, the knowledgebase mainly comes from The video clip of the first talking-head video comes from Liz Bellward, Australian Zoologist (, who loves Big Cat!

Share this project: