Le Khan Academie

Let finetuned Mistral 8x7B generate khan academy videos on demand.

Comment

Watch our other youtube video to see several examples of our finetuned Mistral 8x7B generated education animation videos - Link

Inspiration

  • Inspired by Khan Academy's impact on accessible education worldwide. We wanted to mimic his style of education and make it even more accessible, personalized and on demand. So we created an app for generating high-quality, educational videos on-demand using Mistral models.

What it does

  • Le Khan Academie allows users to input queries (e.g., "How does the distributive property work?"). Then Mistral large rewrites this query with more details on what the manim animation should include. This rewritten prompt is then passed into the finetuned Mistral model.
  • Generates a Khan Academy-style video explaining and visualizing the concept with:
    • Animation created with finetuned Mixtral 8x7B on Fireworks AI that generates python code using manim animation library.
    • Narration by a voice clone of Sal Khan, synchronized with the animation for a brief lecture.

How we built it

  • Utilized the Mistral Large LLM to rewrite user queries into detailed animation specifications using fewshot prompt engineering.
  • Used a fine-tuned Mistral 7x8B model, trained on Manim code outputs, to generate Python code for animations.
  • Employed Eleven Labs' voice cloning technology for creating a Sal Khan voice clone for narration.
  • Synthesized a lecture script, generated narration audio, and synchronized it with the animation.
  • Displayed the final video on the frontend for educational viewing.

Challenges we ran into

  • Some of the manim animations were funky for the more complicated topics in math, especially when it involved a lot of text.
  • Finetuning took some time for Mistral 8x7B.
  • We noticed the manim generations by Mistral were pegged to an older version of Manim so we had to resolve some of the issues that came with that.
  • Synchronizing narration with animation for a seamless educational experience.

Accomplishments that we're proud of

  • Developed a tool capable of generating high-quality educational videos on various topics in math, physics, chemistry, statistics etc. See our youtube video for many examples that our finetuned model generated: Link
  • Enhanced students' learning experience with visual animations and clear narrations in Khan Academy style.
  • Curating training data for finetuning (we ended up using 300+ hand curated examples) and tweaking prompts for simple yet informative animations.

What we learned

  • Gained experience with fine-tuning the mistral models, creating a great finetuning dataset, prompt rewriting, using elevenlabs etc.
  • Learned effective model prompting, fine-tuning, and integration of AI components into educational tools.

What's next for Le Khan Academie

  • Expand topic and subject coverage. Improve finetuning dataset to include more and diverse examples.
  • Increase length of education videos.
  • Introduce more elements of interactivity for the student.

Built With

Share this project:

Updates