We were motivated by two observations. On the one hand, education is siloed into classes with little intentional overlap taking place between them. This represents a failure of modern education since different courses are rife with intersections and when seen, can augment a student’s experience. On the other hand, language models have reached a point where they can effectively process and synthesize information.
By priming a large language models on previously seen content, we can build a personal librarian that can query and explain previous references to a concept. Thus, breaking down course-walls and creating one learning environment.
What it does
We first processed all video lectures and slides for four courses converting video and PDFs to text. Then a user is able to upload a new slide or PDF of a reading they are consuming. User’s can then highlight specific text and Nostoi will describe what the term means in the given context. Next, a user can ask when they have previously seen this term and it will point to a previous lecture where they have seen that content, and summarize the last context where they have seen this.
How we built it
We first downloaded all lectures for four courses using the Switchtube API. We then converted video -> audio and used OpenAI’s Whisper to extract text from audio. Similarly we converted all PDFs to text using a Python library. On this content we built an Elasticsearch to query mentions of a specific term. We built the frontend using Blazer and a few libraries to convert a PDF upload, highlight functionality, as well as an interface for a chatbot. The chatbot was built using GPT-3 and prompt-matching.
Challenges we ran into
PDF highlighting is not trivial. Designing correct prompts for GPT-3 is not easy.
Accomplishments that we're proud of
A working demo that is able to query historic slides and lectures. An assistant that is able to understand intent of a question.
What we learned
Python is not great for backend.
What's next for Nostoi - Your Personal Librarian
- We hope to partner with EPFL or ETH to implement and test this project.