Because of the COVID-19 outbreak, our institution has moved to online teaching for a few months and now online teaching becomes the worldwide hottest topic for all schools!

To be honest, none of the online teaching solutions meet the real need for running online lab sessions. All existing online teaching solutions are just for online-meeting or one direction broadcast! Teachers are 100% no way to know your students are really working on their exercise at home or they are just open the online-meeting apps, and then doing other stuff!

On the other hand, most of the students are not willing to ask questions during an online class. The main difference between online class and face-to-face class is there is lack of interaction. Without classmates, buddies, or young students feel bored and alone then they do not join the online class!

What it does

1.View and Capture All student Screens

Once teacher clicks on the “Generate 3 hours Screen Sharing Tickets”, all virtual tutors (Amazon Sumerian host) will ask students to share their screen to teacher at the same time, then teacher can view all students' screens. In fact, teacher can send the instruction to a virtual tutor and it will talk to all students at the same time. If teacher clicks on the individual student screen, it will enlarge the student screen, and teacher can send a private message to that student.

** This is a very important feature as now teacher can have an overview of the whole class and know which student is really working on their exercise his own. As a result, our online class attendance is dropped as students cannot just open the online-meeting apps and go way or doing something else. However, students who attend the class are sure they are really learning something at home! **

Teacher View from one of our lab class demo


All Screens - Share Sharing and Virtual Tutor Class communications demo


Explain how to use it without the voice from Sumerian demo


2.Virtual Tutor Multi-Language Subject Specific Question and Answer

Teacher response the question from students: teacher just need to upload their lecture notes to Amazon S3 bucket and create an index in Amazon Kendra - Highly accurate enterprise search service powered by machine learning, then Virtual Tutor can answer subject related question from all students with the brain from Amazon Kendra. Students ask a question: Students can ask English questions to the Virtual Tutor with Mic. If they want to ask the non-English questions, students can type in the question and click “Ask” button.

All Screens - Amazon Kendra as Brain Demo


Amazon Kendra answering 3 questions in different languages:

What is web application?

ORMとは何ですか?(What is ORM?)

什麼是面向對象? (What is object oriented?)

3.Virtual Tutor Chit Chat

Student talks anything with Virtual Tutor and she will show different gestures based on her answer. Students like to play game and have fun, so a bit of gamification element is good for any class!

** All Screens - Chit Chat (Flirt with the Virtual Tutor!) Demo **


Demo conversation:

I am sleepy

do I look good

I'm so lonely

I love you


je t'aime

4.Work alone mode

In some cases, we have to do an online quiz or test. Capturing student’s screen is not enough as they might find someone and work together. Therefore, all screens support Webcam Face detection.

privacy of Face detection: we do not upload the student image to AWS and the face detection done inside the student’s browser.

All Screens - Webcam Face Detection Demo


How we built it

Overall System Architecture

Alt text

1.View All student Screens

Alt text

Sending 3 hours ticket to students: uses Cognito User Pool with 2 user groups – teachers and students. By default, the “students” group has granted the IAM permission to upload screen capture image to Amazon S3 bucket, but it cannot limit the number or the valid upload time easily, so the system uses a ticket to control when students can share screen.

Sumerian generates the speech for the host: uses AWS AppSync to connect all teachers and students together in real-time and the communication between Amplify React Apps and Sumerian Host is just using “window.postMessage”. Logic is moved to React but not the state machine of Sumerian host, which is the main difference from Lab Monitor, and we can version control it easily. Once Sumerian host receives the message, Amazon Polly generates the speech for the host.

2.Virtual Tutor Multi-Language Subject Specific Question and Answer

Alt text

For the voice input from students: when a student holds the mic icon, Amplify React Apps calls getUserMedia API to get the voice stream. Amazon Transcribe converts voice stream into text message, and text message sends to Amazon Lex Chatbot. if the Intent name is empty when the question can match defined intent and return from error handling, it means that is not a chit chat. We know student is asking a subject related question.

If teacher sets the Amazon Kendra Index for his class, the question sends to API gateway REST API based by AWS Lambda. It checks the answer from the Amazon Kendra Index which indexed the notes of this subject and response the answer to Amplify React.

AWS JavaScript client SDK does not support querying Amazon Kendra directly, so we have to add a REST API to proxy the query. Sumerian host receives the answer from “window.postMessage”, it converts the text to speech with Amazon Polly.

Alt text

For text input: the flow is similar to the voice input flow but the question from the student first sends to Amazon Comprehend which detects the dominant language of student question and translates it into English with Amazon Translate. The English text message sends to Amazon Lex Chatbot and response message shares the same flow as voice input.

3. Virtual Tutor Chit Chat

Alt text

Chit Chat: the difference is that Amazon Lex Chatbot will return an Intent Name and a response message. Actually, we added 86 common Intents for small talk, so this Amazon Lex Chatbot can do a chit chat. The response message will send to Amazon Comprehend for sentiment. According to the sentiment, Sumerian host will show different gestures. If the question is not in English, the response message translates back to the question’s language with Amazon Translate and display the answer in English and the question’s language. The English response message sends to Sumerian host, and the host specks out the answer with Amazon Polly.

4.Work alone mode

Alt text

Face detection: captures webcam images every second. We use face-api.js which provides face detection API and Machine Learning model for TensorFlow.js. After the face detection, face highlighted image sends to Sumerian and it displays in the projector of the virtual lab. When there is more than one face, React sends a warning message send to Sumerian host, and the host specks out the answer with Amazon Polly.

Challenges we ran into

1.Node.js React Library conflicts.

2.Source Version of Amazon Sumerian.

Accomplishments that we're proud of

By combining the power of each AWS Services, now teacher can have a good view to know each student what they are doing during the online course.

The initial version is already being applied to real online lab lessons in Hong Kong Institute of Vocational Education (Lee Wai Lee), and improved the learning and marking efficiency.

What we learned

  1. The integration with multiple AWS AI Services.

  2. The integration of Amazon Lex, Amazon Comprehend, Amazon Sumerian, Amazon Translate and Amazon Polly for chit chat.

What's next for All Screens

Try to use Amazon Chime SDK to make a better screen sharing view with WebRTC.

Built With

  • amazon-comprehend
  • amazon-kendra
  • amazon-lex
  • amazon-polly
  • amazon-sumerian
  • amazon-transcribe
  • amazon-translate
  • amazon-web-services
  • appsync
  • aws-amplify
Share this project: