About

“Technology will never replace teachers, but a teacher who cannot teach with technology will be replaced by another one who can.” – Zuzana Molčanová, Teacher Engagement Managers, Microsoft

One of the many impacts of Covid-19 has been the dramatic acceleration and reliance on technology for education.

Although the change to teaching online has felt forced by many some are embracing the change, technology has the potential to not only improve the way education is delivered, making lives easier for students & teachers but to have a big impact on improving learning outcomes.

With the right tools and technology online education can be more effective than offline/traditional methods.

I work for LearnCube a company focused on delivering effective live online classes, through our online school platform and Virtual Classroom software.

We currently help educators deliver over 100,000 live online classes each month, integration with Microsoft Teams has been a common request so this hackathon was the perfect opportunity to integrate LearnCube's Whiteboard API into Microsoft Teams.

I also took the opportunity to release our new StudyBot and A.I powered teacher assistant tools.

What it does

The LearnCube Whiteboard & Teacher Assistant app adds an advanced whiteboard designed for online education for use within Teams. The whiteboard allows real-time collaboration of lesson plans between teachers & students. Participants can interact with content using a range of education-focused whiteboard tools.

One of the main features in this integration is the ability to highlighted words & sentences during live classes on Teams, we then provide content suggestions in real-time to use within the class. Highlighted words & sentences are also stored as part of the class review with then can be easily converted into interactive content & quizzes.

We also provide a OneNote Add-in that allows Teachers to easily share content between the Teams the LearnCube Whiteboard, and OneNote.

Features:

  • Advanced interactive/collaborative whiteboard designed for Teaching online

  • An easy way to upload/share many common file formats of lesson plans and material

  • We can automatically make static content (PDF's/Images) interactive by using a custom OCR to identify "Fill in the gap", "checkboxes" and "answer fields" within the content.

  • Post-class review allows students to access class content and homework tasks easily for any past classes.

  • A.I generated quizzes, easily convert class notes into interactive exercises that can be personalized per student, based on their own notes.

How I built it

FrontEnd

VueJs, Canvas Library, H5P, Cloudinary

Backend:

Python, Django, Open-CV, Wolfram Alpha, Spacy.io, Node.js, Azure Bot

I used LearnCube's existing Whiteboard API to integrate directly into Teams.

This is also the first release of our Azure-powered StudyBot that uses NLP/NLU (MS LUIS & QnA Maker) we use several 3rd party sources for the data including Wolfram Alpha, Wikipedia, YouTube, Datamuse as well as our own data and corpus sets, we created custom Microsft Adaptive cards to display the recommendations within the Azure Bot.

Our custom OCR model is used to extract lesson content data and activity type

Trained and labeled content /input/activity types from 1000 of different lesson plans and materials so we can effectively: summarise a lesson plan (per student level) identify the subject level and approx grade/difficulty, identify which input types to convert into digital inputs.

This data allows us to make relevant content suggestions for use in the class in real-time, convert static content into interactive content and create interactive quizzes from keywords and phrases.

Microsoft APIs used:

  • Azure Bot Builder
  • LUIS
  • QnA Maker
  • Bing Search
  • Bing Entity Search
  • Bing Answer Search
  • Translator
  • Speech to text
  • Text to speech
  • Adaptive Cards
  • OneNote
  • Teams

Challenges I ran into

Custom Adaptive cards with carousels seem to be not supported for Bots inside teams, so I integrated separately.

Developing reliable custom OCR and NLPNLU intents that work with a range of different publisher/format activity types is an ongoing challenge that will improve over time with more data.

Currently not all formats or activities get auto-detected by the A.I model so also added a manual way for teachers to edit/add their own interaction which also helps as "labeled" annotated data for the A.I model to "self learn" (CNN “self-supervised learning") so over time it will get better at identifying more different content types.

The App needed to be very simple and intuitive to use to balance that with a powerful/useful A.I tool that enhanced the learning and environment without adding distraction was also a challenge but happy with the end result.

What's next

Adding a teacher admin panel to easily manage class schedules within Teams and a dedicated UI flow for creating synchronous content tasks and homework.

OneNote is a very popular tool for Educators so further integration between Teams, LearnCube, and OneNote would make teacher and student workflow easier.

Adding paid premium plans which include license content and lesson plans and more quiz types and interactions.

This is our first officially Microsoft product integration so we are eager to get feedback from educators and students to see how else we can improve online education.

Being part of the Microsoft Teams App store will help us reach more educators who can use our integration to deliver even more engaging and effective online classes.

Share this project:

Updates