hands on

Laboratory equipment are expensive. Learning on 2D blackboard is unengaging. Hands On is AR simulation equipped with AI for both teachers & students to interact, experiment, and make mistakes freely.

Comment

Team member: Tung Viet Le

Inspiration

I remember the first time I went to my physics lab during my high school years to experiment with electronic circuits. The equipment looks nothing familiar to me even though I have been studying it for a year. Turn out I was working with 2D symbol drawings the whole time, without touching or experiencing myself.

What it does

Laboratory equipment are expensive. Learning on the 2D blackboard is unengaging. Making Physic Experiment accessible anywhere, Hands On is an AR simulation for both teachers and students to interact, experiment, and make mistakes freely. It makes classroom experience more engaging for everyone. Better yet, Hands On is equipped with AI to instruct, explain to students in real-time: like "You should turn off the power source before plugging in", or perform actions like "Give me the set of equipment for Archimedes experiment", which can enhance creativity, safety, and familiarity.

Interacting with the environment using natural language

Speak to Hands On

  • “I need a weight[object] made of iron[material] to conduct a buoyancy force experiment”.
  • “Change the simulation gravity[field] to the moon’s gravity[value]”.
  • “Set the height[field] of liquid[target] to 57cm[value]”.
  • “Spawn something to measure length[description]”.

Best Practices:

  • i. It’s good to keep it concise about what your intentions are.
  • ii. You can ask Hands On to assist you with details. Example: “Show me the forces analysis of objects.” - iii. Beside speech interaction, you can also interact with the simulation using real-world physical movement. Example: “grab object, move object around.”
  • iv. Users are also able to take notes anywhere in the environment, so the world is your infinite, limitless canvas.

How I built it

Input utterances, label data, train, test, improve, and deploy AI model using Azure CLU. Connected Cognitive Search and SQL database. Code 3D physic simulations from scratch using Unity C# Develop for Android AR Core

Challenges I ran into

Deciding entities for labeling, model overfitting performance, SDK dependencies installation, connecting APIs, implement proper simulation environment.

What I learned

It was a rewarding learning experience for me. I learned lots of new concepts and technical skills. I'm looking forward to collaborating with students, professors at my university to receive feedback and mentorship. I'll keep developing, and updating this project further.

What's next for hands on

  • I have seek professor mentorship and found ML students who are interested. We can turn this into testing in UMass Amherst by Fall 2024.
  • Real-time feedback
  • Multi-User Collaboration on the same AR environment
  • Tool recognition using computer vision
  • IOS ARKit development

Built With

  • argumented-reality
  • azure-ai
  • azure-cognitive-language-understanding
  • azure-vision
  • by-voice
  • c#
  • sql
  • unity
Share this project:

Updates