AdaptEd

interactive and personalized lectures, through conversational voice AI

Comment 1

As a K-12 tutor for Kumon, one of our teammates realized the massive impact that personalized, 1:1 tutoring had on students. However, not everyone has access to enrichment programs, creating a compounding limitation that affects success in future endeavors.

Out of 16 million university students in the United States, 50% of them are falling behind due to static and one-sided teaching without personalized engagement. At the same time, less than 3% of students have access to quality tutoring programs.

AdaptED turns the educational paradigm on its head: instead of students adapting to the system, our ai lecturer adapts to students. Through real-time conversation, verbal/visual content changes in real time.

What it does

AdaptED turns lectures into conversations with a live humanoid lecturer that a user can speak to directly. Corresponding lecture slides are generated live based on the user's verbal response, creating an adaptive learning experience.

It performs 3 main functions:

  1. responsive ai: user can conversate with the lecturer
  2. dynamic content: slideshow and whiteboard content changes based on verbal response
  3. emotion detection: measures the user's attention and confusion

How we built it

  • Multi-source Aggregation Pipeline:
    • Gemini 1.5 Pro: Huge context length allows us to aggregate truly enormous amounts of data from various sources:
      • Audio: Youtube Videos, Podcasts
      • Video: Youtube Videos, Lecture Recordings
      • Massive Amounts of Text: Textbooks, Wikipedia Articles
  • Action Taking Dynamic Agent:
    • Interruption and End of Turn Detection allows for natural conversations.
    • Action taking lets agent modify slides and whiteboard in real time according to user's queries.
  • Intel Developer Cloud:
    • Allowed us to fine tune an open source model to gain more accurate responses
    • Acting as a real time inference engine for fast generation
  • Fetch.ai
    • Agents that can perform tasks on the side without interrupting main speaker
    • Proxy Queries allows agents to automatically activate when users perform an action.
    • On message callbacks allows the main speaker to continue speaking while waiting for information.
  • Google Search: allows us to add both images and additional dynamic information to our slides.
  • MongoDB: lets us store lectures to be viewed at a later time.
  • Auth0: lets us save user trends and statistics for each user'
  • Hume: Real time emotion detection can be used to tailor responses.

Challenges we ran into

  • It was tough learning how to use new platforms such as Fetch.AI and Intel Developer Cloud (IDC)
    • Especially when the documentation was sparse and confusing.
  • Hardware limitations meant custom software and models could not perform as best as they could have.

Accomplishments that we're proud of

What we learned

  • Hardware is cool
  • How to design multidimensional agentic systems
  • To look at example notebooks before doing my own

What's next for AdaptED

  • Add more agents, easier scaling
  • Add more actions
  • Google doc integration
  • Personal history, statistics visualizer

Built With

+ 1 more
Share this project:

Updates