Sheldon

Inspiration

Imagine yourself sitting in a class. It's a difficult course and you probably made a mistake taking it. You need help understanding what is going on in class and absorbing a lot of things (aka prerequisites) before you can start to comprehend what the professor is saying. It can happen to the best of us and we believe that we have built an AI tool for when you get into a sticky situation like that.

What it does

Sheldon, the AI interface we built, is attending the lecture alongside you and providing you with the information that might come in handy. You can see answers to the questions that might come up in your head, see visualizations related to the topic (because who isn't a visual learner?), and you also get to ask it questions about the lecture so far to bring you up-to-speed on the lecture and feel confident in your ability to take on any challenging course.

How we built it

Our multi-media pipeline is watching the lecture using DroidCam on your phone. It can see the video stream and extract key slides / frames and audio to create a transcript of the lecture using Google Gemini. It processes it further to identify difficult concepts, opportunities for visualizations, and for chat context for your important questions, again using Google Gemini. The visualizations are found using Google's Programmable Search Engine. The front-end is built in ReactJS and backend in Flask. These keeps a track of the media stream and push the information to users browser window. Other than these, it took 17 RedBulls, 3 Celsius, 4 people and 2 sleepless nights, and we are proud of what we have achieved.