Magic Leap, Unity, IBM Watson Unity Plugin (Speech-to-Text), Free models from Unity Asset Store.


Modern schools are still based upon an old factory paradigm that struggles to meet the individualized needs that each student deserves. Schools won't go away anytime soon. Student variation will never go away. With In. Class, it is the boring, uniform classroom that goes away.

We started with a North Star of helping english learners. Research shows the students learn a language best when shown closed-captions below a video. We wanted to add this to real-life transcription to classrooms. Once we brought that technology to the student, it opened a rich world classroom possibilities. IN.CLASS is accessible by providing closed captioning, by tracking focus, and by facilitating truly differentiated learning.

IN.CLASS is an immersive in-classroom tool for students and teachers that helps with focus and learning. By providing unique tools, such as closed captions and 3D visual aids, it increases students comprehension and eases the teaching burden. While teachers can review the engagement level of the students, real-time transcriptions of the course support english learners. 3D tools let teachers push 3d objects to helps everyone better learn the material. Differentiation lets a teacher push, say, a 3D heart to each student's desk, but each students may receive different models of varying complexity, depending upon their own learning plan.

What it does

IN.CLASS is a platform that helps both the teacher, and the student.

IN.CLASS brings augmented reality to the classroom, and gives teachers a control panel to monitor student progress, follow lesson plans, and bring Mixed-Reality-aware tools to their fingertips.

Students see their desks transformed from a boring slab, into a dynamic workbench. The student can receive pushed learning-objects from the teacher. Items can range from simple quiz questions, to personalized learning aids, such as 3D visual aids and experiential simulations.

Feedback tools, such as automatic attentiveness tracking, give teachers ways to more easily tailor the the class in real-time.

Technology Differentiation

  • Real-Time closed-captioning of the teacher
  • Teacher initiated on-desk quizzes
  • Teacher initiated 3D objects to the desktop
  • Attention tracking of students, relaying to teacher

Value Differentiation

  • Built-In Differentiated Experiences (The teacher could push a 3D heart to the class, and AP students would receive a more detailed model, but standard student could receive a less details model appropriate to their own learning plan).
  • Each student has a different set of needs, and can receive a unique experience, depending upon their situation.
  • English Learners can receive real-time transcription
  • Deaf students could read transcription, or potentially pipe-in real-time signing services

  • A large class demands much from a teacher, our Teaching Control Panel helps

  • Instantly track attention and/or confusion (via eye tracking) and tailor the lesson, accordingly.

  • Quickly and dynamically push 3D visual aids to students

How we built it

We're using the Magic Leap to create a mixed-reality environment for the student. Unity is our engine, and IBM Watson provides our real-time transcription services.

Challenges we ran into

None of us had worked with the Magic Leap before, so a lot of time was with system basics. We explored the capabilities of the system and spent way to long getting the dev tools installed.

Accomplishments that we're proud of

The transcription concept works surprisingly well.

What we learned

Brainstorming with a diverse group can lead to great new ideas.

What's next for In. Class

IN.CLASS is a fun proof of concept that shows a possible future for the classroom, but the underlying idea of the teacher-student connected platform is a powerful concept that could implemented with existing technologies and budgets. Our team hopes to incubate the concept into a practical product.

What would a one-year product look like? Imagine starting with Magic Leap classroom as the vision, but targeting classes with Smart Boards and iPads, which are in place today. A teacher you could still pushing quizzes, questions, and 3d objects to each iPad. Objects and questions could still be differentiated based upon the student. Teachers could project a room-sized object and students would lift their iPad to see an augmented view of the room, seeing that room-sized object.

Unrelated to learning objects, realtime transcripts, pushed to an mobile phone or to simpler over-the-eye displays (like Google Glass, or Epson Smart Glasses), could be implemented and tested in-class within the year.

Built With

Share this project:


posted an update

Subtitles - Risk Mitigation

We've wanted to add subtitles (closed captioning) to the classroom teacher in our environment from the beginning, but we were concerned about the feasibility. I'm happy to report that we've been able to add an initial proof-of-concept into the platform using IBM-Watson for speech to text translation.

Anyone working on their english, or just suffering through an instructor with a too-strong accent, will appreciate this new feature.

Log in or sign up for Devpost to join the conversation.