G(Y)AN.AI

GYAN.AI Landing Page

Video Demonstration: https://www.loom.com/share/da318c874b064374b10cfaa06bc8f6cb?sid=f35a5226-aac7-4f23-a766-27a3411cf535

Inspiration

We were inspired to create an accessible study helper observing the challenges our friends face. My (Anupam's) girlfriend has learning disabilities, and I have seen first-hand how much time she spends rewatching lectures and trying to create notes and get study help. We have also noticed the lack of note-takers in our classes and how we have been requested to take notes for classes to help students for accessibility.

What it does

Considering these challenges, we decided to build an AI web-app (GYAN.AI) that provides personalized study tools to students with learning disabilities. GYAN.AI is the first video-analyzer AI geared towards students. Students can upload a video to our web-app and get concise and structured notes, study plans, summaries, and use active-recalling strategies by attempting quizzes. GYAN.AI also supports multiple languages, allowing students to access study material in their own preferred language as text or speech.

How we built it

Backend: Our backend consists of 6 Flask microservices that interact with each other through routes to create sequential LLM chains, fine-tuned to accurately analyze videos and generate study tools geared towards students with learning disabilities.

AI Model-training:Google T-5 LLM was fine-tuned for the summarizer functionality to reduce its creativity & Mistral-7B-v01 was fine tuned as well to provide better quiz functionality & accuracy. These LLMs were fine-tuned by Supervised Training using informational & news datasets. Pipelines & transformers were used to augment the LLMs such that their answers become more oriented towards the input provided.

Frontend: Our frontend is built using React with Vite. We use the NextUI library for our components, and we have created custom components using Tailwind CSS as well. We have multiple custom-UI components that are embedded to our website and communicate to each other through arguments and local JSON files. The frontend communicates through the backend using Axios requests.

Challenges we ran into

Implementation with SupaBase: We were hoping to implement our app in SupaBase. We could only connect SupaBase to 5 of our 6 micro-services. Since our micro-services were connected to each other, we found it challenging to make them work together. We decided to scrap the idea today morning as we wanted to focus on adding finishing touches and accessibility features to the app.

Incompatibility of our devices to fine-tune models locally: We could not use pipelines and transformers on our Macbook as we needed an AMD-x64 type chip to run them.

Playing sound on our website: We found playing sound on our website (for the speech in multiple languages) functionality hard. We were getting hit by browser guardrails, and could not end up playing speeches on our website. We were very close to solving this issue, as we could play locally stored audio (in the same directory). We decided to keep the functionality to implement in the future as we wanted to focus on finishing up the project.

Accomplishments that we're proud of

We've built the first video-analyzer AI app geared towards students with disabilities and from diverse backgrounds.
We are proud of the accuracy of our fine-tuned LLM models, which worked way better than other counterparts.
We've strived to build an accessible user-experience, and we are proud of how seamless, intuitive, and modern our UI is.

What we learned

Always start with the most difficult sections when you have the maximum resources available. We started our project with the Video to Notes microservice, which was technically challenging.
The importance of clear division of workload amongst team members. We were able to split up tasks amongst the 3 team members efficiently, and that helped us implement so many functionalities.
Don't push your API keys on Github. Please.