Orbit

Education that thinks. Watches. Adapts. We gave your study materials a brain. Orbit tracks your gaze to decode your doubt, morphing the content in real time until the concept is yours.

Comment

Orbit: The Intelligent Adaptive Learning Platform

Inspiration

Education isn’t one-size-fits-all, but most learning tools still are. Learning with AI today often means getting stuck in repetitive prompt cycles --> ask it to explain a syllabus, then teach a topic, then generate questions, then answer overall just wasting time instead of actually learning. Orbit was born from the idea that a study companion should adapt to you automatically, not just in pace but in confusion, confidence, and engagement. We wanted something that feels less like prompting a chatbot and more like a perceptive tutor that knows when you’re drifting or when a concept isn’t clicking.

What it does

Orbit transforms static learning materials into a living, breathing educational ecosystem.

  • Smart Decomposition: Upload any PDF or text, and Orbit parses it into structured modules and subtopics.
  • No Prompt Loops: Orbit eliminates repetitive prompting cycles by handling teaching, questioning, evaluation, and revision in a single continuous learning flow.
  • Adaptive Lessons: Each subtopic becomes a personalized lesson with rich content blocks text, formulas, and most importantly, interactive simulations; Google Gemini 3 dynamically generates explanations, analogies, and simulation scaffolding tailored to the detected difficulty of each subtopic.
  • Real-Time Engagement Tracking: Using your webcam (processed primarily client-side for privacy), Orbit analyzes your facial expressions to gauge real-time confusion or confidence levels.
  • Dual-Score System: We combine your quiz performance with your engagement metrics to generate a "mastery score" for each topic.
  • Targeted Revision: Orbit knows what you don't know. At 25%, 50%, 75%, and 100% curriculum completion, it dynamically generates revision modules specifically targeting your weak spots.
  • Accessibility First: A fully voice-navigated mode for visually impaired users. Toggle it with Ctrl+Space, hold Ctrl to speak, and navigate the entire app including answering questions without seeing the screen.
  • Dynamic Performance: The frontend adjusts visual quality based on live FPS checks to ensure smooth performance on any device.

How we built it

Frontend

  • Core: React (Vite), TypeScript
  • Styling: Tailwind CSS, Shadcn/UI (Radix Primitives)
  • Animations: Framer Motion for smooth transitions
  • 3D Graphics: Three.js (@react-three/fiber) for immersive visuals
  • State Management: TanStack Query for caching and synchronization

Backend

  • Framework: FastAPI (Python)
  • Authentication: Clerk (@clerk/clerk-react)
  • Database: PostgreSQL (Neon Tech) with SQLAlchemy & Psycopg2

Artificial Intelligence & Processing

  • LLM: Google Gemini (google-genai) for content generation
  • NLP: Spacy for natural language tasks
  • Computer Vision: MediaPipe for real-time face tracking
  • Text-to-Speech: Piper TTS (backend) for efficient voice synthesis
  • File Parsing: Unstructured for robust document ingestion

How Gemini Powers Orbit

  • Gemini 3 acts as the cognitive engine behind Orbit. Instead of being used as a simple text generator, it is leveraged for its long-context understanding to reason over entire documents at once.
  • Gemini 3 produces adaptive explanations that adjust based on learner performance. As users interact with lessons and assessments, revision content is regenerated dynamically using performance signals instead of fixed templates.
  • Google AI Studio (Gemini) was also used during the planning and ideation phase to design Orbit’s system architecture, learning flows, and adaptive logic, helping structure the project end-to-end before implementation.

Challenges we ran into

  • The Parsing Bottleneck: Initially, Unstructured was taking far too long to parse files to be viable for real-time user uploads. We researched deeply, reached out to their engineering team, and optimized our pipeline to cut parsing time by over 50%.
  • Computer Vision Performance: We initially implemented face tracking using MediaPipe, experimented with TensorFlow.js as an alternative, but encountered significant performance issues that severely impacted frame rates. We ultimately reverted to MediaPipe, which delivered stable performance and improved FPS by approximately 78%, keeping the UI smooth.
  • 3D + Motion: Integrating Three.js canvases with Framer Motion animations was a steep learning curve. Getting them to play nice together required a lot of trial and error (and coffee).
  • Simulation Struggles: Generating interactive simulation blocks that were both accurate and educational was tough. We had to rework the prompt engineering and rendering logic multiple times.
  • Voice Gremlins: Configuring Piper TTS in the backend to be reliable and responsive was a surprisingly thorny issue that required deep debugging.
  • API Reliability and Rate Limits: During testing, we frequently hit API key exhaustion limits causing request failures. To address this, we implemented an API key routing layer that distributes requests across multiple keys (five in total) to sustain throughput. Additionally, we added a fallback mechanism to automatically switch to Gemini 2.5 whenever the primary model failed, ensuring the application remained stable and did not crash.

Accomplishments that we're proud of

Honestly? The sheer integration of it all. We have a Python backend talking to a React frontend, real-time computer vision running alongside 3D graphics, a custom voice engine, and an LLM orchestrating the content and it all works seamlessly.

The Visual Impairment Mode is a personal highlight; navigating a complex app purely by voice feels like the future of accessible tech.

This vision became feasible due to Gemini’s long-context understanding and structured generation capabilities. Rather than freely generating responses over an entire document, Orbit uses Gemini to remain tightly scoped to the active topic and its source material, preserving the original content structure and context. This grounding significantly reduces hallucinations while enabling accurate, focused explanations.

What we learned

  • We learned that "state-of-the-art" often means "you have to figure out how to make it work yourself."
  • We learned how to efficiently optimize workflow for a large code-base such as ours with the proper use of git/github .
  • We learned several techniques to optimize functioning of the back-end system and to get better performance in practical use-cases.

What's next for Orbit

  • Multiplayer study sessions
  • Deeper gamification mechanics
  • To create an Android App

Team

-Shivansh Verma -Raj Somvanshi

Note: Two additional contributors are listed on GitHub; however, they were involved only during the initial ideation phase and did not contribute to the final implementation.

Built With

Share this project:

Updates