🌟 Omni Tutor MCP: 3D Interactive STEM & Coding Studio
💡 Inspiration
Learning shouldn't feel like reading a manual.
Yet that's exactly what happens with most AI tutors. Ask an AI to explain physics, mathematics, or programming, and you usually get a wall of text. Ask the AI to check your code, and it might confidently tell you that broken code is correct.
We wanted to build something different.
What if an AI tutor could become a laboratory instead of just a chatbot?
That idea led to Omni Tutor MCP — an interactive STEM and coding studio where students can experiment with physics, mathematics, algorithms, audio, and code directly through interactive 3D environments.
Instead of simply telling students the answer, Omni Tutor lets them see it, manipulate it, test it, break it, and discover why it works.
At the same time, we wanted to solve the reliability problem of AI tutoring. The AI should be able to explain and guide, but it should not be trusted to determine whether a student's answer is actually correct.
So we created a deterministic verification layer underneath the AI.
The result is a tutoring system built around three principles:
- 🎮 Interactive learning instead of passive reading
- 🧠 Socratic guidance instead of answer dumping
- 🛡️ Deterministic verification instead of AI-based grading
🚀 What it does
Omni Tutor MCP turns an AI conversation into an interactive STEM laboratory.
Using Model Context Protocol (MCP) and Generative UI, it can dynamically bring interactive experiences directly into the learning environment.
Students can:
- 🌍 Change gravity between Earth, Moon, Jupiter, and Zero-G
- 🚀 Launch projectiles and experiment with physics
- 🧱 Explore collisions, momentum, torque, and energy in a real-time 3D physics world
- 📐 Manipulate 3D mathematical surfaces and gradient fields
- 🧊 Watch algorithms execute through animated 3D voxel visualizations
- 🎹 Play sounds and experiment with waveforms through an interactive synthesizer
- 💻 Write and preview HTML, CSS, and JavaScript in a live sandbox
- 🧑💻 Solve coding and physics missions
- 🧠 Receive Socratic hints instead of having the answer handed to them
- 🌌 Explore AI-generated 3D scientific and astrophysical scenes
- 📊 Track learning progress and mastery
Behind the scenes, the system contains 13 MCP tools and 6 interactive Generative UI experiences.
But the most important part isn't the number of tools.
It's the philosophy:
Don't just explain the concept. Let the student experiment with it.
🛠️ How we built it
We built Omni Tutor MCP as a bridge between AI reasoning, interactive 3D environments, and deterministic verification.
The backend is powered by Python 3.12 and the MCP Python SDK, communicating through Streamable HTTP.
The interactive experiences are built with TypeScript, Three.js, Cannon-es, Web Audio API, and @modelcontextprotocol/ext-apps.
Three.js gives us the 3D world.
Cannon-es gives that world real-time physics.
The Web Audio API turns the browser into a musical laboratory.
Generative UI connects everything back to the AI conversation.
And Gemini acts as the tutor.
But we didn't want Gemini deciding whether a student's answer was correct.
So we built something more important:
🛡️ The Deterministic Trust Boundary
Student code is executed in an isolated Python subprocess and checked against strict unit tests.
Physics predictions are validated using actual mathematical equations.
The AI sees the results and uses them to guide the student.
In other words:
Student
↓
AI Tutor
↓
Real Experiment
↓
Deterministic Test
↓
Ground Truth
↓
Socratic Guidance
The AI explains.
The system verifies.
⚡ Challenges we ran into
Making an AI tutor that can actually do things turned out to be much harder than making one that can simply talk.
We had to synchronize a 60 Hz physics engine with real-time Three.js rendering inside a Generative UI environment.
We had to make complex 3D experiences self-contained and lightweight.
We created procedural scientific visuals instead of relying heavily on external assets.
And we had to solve the biggest problem of all:
How do you make an AI tutor helpful without letting it hallucinate the truth?
The answer was to separate intelligence from verification.
Gemini can reason, explain, ask questions, and guide.
But correctness comes from code execution, unit tests, and mathematics.
That separation became one of the core architectural ideas behind Omni Tutor.
🏆 Accomplishments we're proud of
We didn't just build a collection of visual demos.
We built an entire interactive tutoring ecosystem.
🔥 13 MCP Tools
From physics and mathematics to algorithms, coding, audio, tutoring, verification, and analytics.
🎨 6 Interactive Generative UI Experiences
Each experience is packaged as a self-contained HTML application and designed to work directly with the MCP environment.
🧠 AI + Deterministic Verification
The AI can guide the student without being responsible for determining ground truth.
🎮 Learning by Doing
Students can experiment, make predictions, fail, change variables, and try again.
📦 Self-Contained Generative UI
All six interactive widgets are bundled as self-contained HTML applications, making the system portable and reducing external client dependencies.
✅ 15/15 Automated Tests
Our comprehensive end-to-end test suite passed:
15 / 15 — 100%
The tests cover the MCP handshake, tools, UI resources, coding missions, physics missions, verification, AI tutoring, 3D generation, mathematics, audio, analytics, physics, and the live endpoint.
Most importantly, we proved that the complete system works together as one experience.
📚 What we learned
We started by thinking we were building an AI tutor.
We ended up learning that AI tutoring isn't really about answers.
It's about creating the right environment for someone to discover an answer.
We learned that:
- Interactive experiences can make abstract concepts tangible.
- AI is more useful when it guides rather than gives everything away.
- Deterministic systems are essential when correctness matters.
- MCP can connect AI models to completely different kinds of experiences.
- 3D visualization can turn abstract scientific concepts into something students can manipulate.
- The best educational AI may not be the one that knows the most — but the one that helps the student discover the most.
The biggest lesson was simple:
The future of AI education isn't just smarter answers. It's smarter experiences.
🔮 What's next for Omni Tutor MCP
This is only the beginning.
We want to turn Omni Tutor into a much larger virtual STEM laboratory.
🧬 Chemistry
Students will be able to build and explore molecules in interactive 3D, learning molecular structures and chemical bonding by experimentation.
⚛️ Quantum Mechanics
We'll bring concepts such as wave-particle duality, probability distributions, wave functions, and quantum states into interactive simulations.
🌐 Collaborative Virtual Labs
Instead of learning alone, students will be able to enter the same virtual laboratory, perform experiments together, and collaborate in real time using technologies such as WebRTC.
🧠 Adaptive Learning
The system will use verified learning history to understand where a student is struggling and dynamically recommend the next experiment or challenge.
🥽 VR & AR
We want to take Omni Tutor beyond the screen and eventually let students step inside their experiments.
🌎 Our Vision
Imagine asking:
"Why does a planet stay in orbit?"
Instead of receiving 500 words of explanation, the AI creates a solar system.
You grab a planet.
You change its velocity.
You increase its mass.
You move it closer to the star.
You watch the orbit change.
Then the AI asks:
"What do you think will happen if we increase the velocity?"
You make a prediction.
The simulation runs.
The result is verified.
And suddenly, you're not just reading about physics.
You're doing physics.
That's the future we're building with Omni Tutor MCP.
Not an AI that simply gives students answers.
An AI that gives students a world to explore.
🚀 Don't just ask AI. Experiment with it.
Log in or sign up for Devpost to join the conversation.