Nostoi - Your Personal Librarian

Inspiration

We were motivated by two observations. On the one hand, education is siloed into classes with little intentional overlap taking place between them. This represents a failure of modern education since different courses are rife with intersections and when seen, can augment a student’s experience. On the other hand, language models have reached a point where they can effectively process and synthesize information.

By priming a large language models on previously seen content, we can build a personal librarian that can query and explain previous references to a concept. Thus, breaking down course-walls and creating one learning environment.

What it does

We first processed all video lectures and slides for four courses converting video and PDFs to text. Then a user is able to upload a new slide or PDF of a reading they are consuming. User’s can then highlight specific text and Nostoi will describe what the term means in the given context. Next, a user can ask when they have previously seen this term and it will point to a previous lecture where they have seen that content, and summarize the last context where they have seen this.

How we built it

We first downloaded all lectures for four courses using the Switchtube API. We then converted video -> audio and used OpenAI’s Whisper to extract text from audio. Similarly we converted all PDFs to text using a Python library. On this content we built an Elasticsearch to query mentions of a specific term. We built the frontend using Blazer and a few libraries to convert a PDF upload, highlight functionality, as well as an interface for a chatbot. The chatbot was built using GPT-3 and prompt-matching.