Use Case Title:
Experiment and interact with Physics Laboratory with Hands On
Description:
We all know that conducting an experiment comes with lots of challenges, laboratory equipment is expensive and inaccessible to most, the unfamiliarity with the tools, and the safety of students, but 2D blackboard is unengaging. Hands On is an AR physics lab app, users can interact with a virtual simulation by requesting using Natural Language, making physics experiments more accessible and safer for everyone. Conducting with friends, family, from students and teachers, to parents and children.
Tutorial for Use and Best Practices:
Interacting with the environment using natural language Speak to Hands On: “I need a weight[object] made of iron[material] to conduct a buoyancy force experiment”. “Change the simulation gravity[field] to the moon’s gravity[value]”. “Set the height[field] of liquid[target] to 57cm[value]”. “Spawn something to measure length[description]”. Best Practices: It’s good to keep it concise about what your intentions are. You can ask Hands On to assist you with details. Example: “Show me the forces analysis of objects.” Beside speech interaction, you can also interact with the simulation using real-world physical movement. Example: “grab object, move object around.” Users are also able to take notes anywhere in the environment, so the world is your infinite, limitless canvas. [Under development] Real-time feedback Anticipated use case: i. “Why is my spectrometer not giving accurate readings?” Real-Time Feedback: Hands On can suggest checking the alignment, the quality of the light source, or the calibration of the spectrometer. ii. "Is it safe to heat this substance with the Bunsen burner?" Real-Time Feedback: Hands On can provide safety guidelines, including the substance's properties and suitable temperature ranges for heating. [Future update] AR Multiplayer in the same environment / Tools recognition using vision Enhance collaboration that allows students to work together and share experiences. Allow teachers to demonstrate how the experiment should be conducted. Import tools into the simulation.
Impacts on Learning:
Positive Impacts: Engagement: AR makes learning more interactive and engaging, increasing student interest in physics concepts. Accessibility: Hands On’s AR and AI can make laboratory experiments more accessible to diverse learners, including those with disabilities. (classroom or at home) Real-world application: Students can experiment with physics in a safe, virtual environment, bridging theory and practical understanding. Immediate feedback: Working solo or with a team, AI can provide instant feedback on experiments, instructing and helping students learn from their mistakes. Collaboration: AR labs can facilitate collaborative learning, allowing students to work together and share experiences. Self learning: AR and AI encourage continued exploration and understanding of physics concepts beyond traditional classrooms. Learning pace: AI can adapt content to individual learning styles, pacing, and needs, enhancing comprehension.
Limitations and Ethical Considerations:
Limitations:
- Lack of Physical Experience: It may not replace real-life experiments in some cases, particularly in labs where physical interactions are crucial.
- Overreliance on Technology: Students may become overly reliant on AR and AI, potentially hindering critical thinking and problem-solving skills.
- Ethical Concerns: Privacy issues and data security concerns must be addressed, especially when dealing with student data.
Link to Video Tutorial (Optional):
https://www.youtube.com/watch?v=0ZqBgqMGOlc
Inspiration
I remember the first time I went to my physics lab during my high school years to experiment with electronic circuits. The equipment looks nothing familiar to me even though I have been studying it for a year. Turn out I was working with 2D symbol drawings the whole time, without touching or experiencing myself.
What it does
Laboratory equipment are expensive. Learning on the 2D blackboard is unengaging. Making Physic Experiment accessible anywhere, Hands On is an AR simulation for both teachers and students to interact, experiment, and make mistakes freely. It makes classroom experience more engaging for everyone. Better yet, Hands On is equipped with AI to instruct, explain to students in real-time: like "You should turn off the power source before plugging in", or perform actions like "Give me the set of equipment for Archimedes experiment", which can enhance creativity, safety, and familiarity.
How we built it
- Input utterances, label data, train, test, improve, and deploy AI model using Azure CLU.
- Connected Cognitive Search and SQL database.
- Code 3D physic simulations from scratch using Unity C#
- Develop for Android AR Core
Challenges we ran into
Deciding entities for labeling, model overfitting performance, SDK dependencies installation, connecting APIs, implement proper simulation environment.
What we learned
It was a rewarding learning experience for me. I learned lots of new concepts and technical skills. I'm looking forward to collaborating with students, professors at my university to receive feedback and mentorship. I'll keep developing, and updating this project further.
What's next for Hands On
- Real-time feedback
- Collaboration on the same AR environment
- Tool recognition using computer vision
- IOS ARKit development
Log in or sign up for Devpost to join the conversation.