RajivAI

Check it out here

Inspiration

As college students, we often struggle with exam preparation due to the lack of practice test material. Because many classes often reuse past exam questions for their exams, professors generally don't release many practice exams for students. Finding online exams can also be hard, especially for harder-level classes, as many colleges curate their own course syllabus. Thus, it is difficult to find an online exam that fully caters to the content of the course you are taking.

From the perspective of professors, creating an exam with a corresponding answer key takes significant time and effort. Professors want their exams to not only be comprehensive but also fit a set of specific constraints.

Introducing RajivAI

To solve both of these issues, we created RajivAI, an LLM exam generator that outputs a sample exam given the course materials and custom instructions. Students can now create unlimited exams to practice with, and instructors can generate exams for students to both practice and actually take.

How it works

RajivAI is built on top of a multi-agent pipeline. Each agent is a GPT-based generative AI that has been prompted and fine-tuned to think and perform certain tasks. We have two different types of agents: Professor Rajiv and TAs.

Using GPT-4, the professor agent is responsible for receiving user prompts, delegating tasks for TAs, and performing executive decisions given the content summarized by TAs.

Supported by GPT-3.5-turbo-16k, TA agents are responsible for analyzing the syllabus to summarize the content and create the exam. These agents are fine-tuned and equipped with external API function-calling to perform mathematical calculations.

We have a single "Professor" agent and 2 TAs per 16,000 GPT tokens (equivalent to around 25 pages of words). The TAs are paired up in groups of 2 to analyze the same chunk of text: one TA generates, and the other TA verifies the answer key using math functions.

For the algorithm to ultimately output an exam, it goes through a multi-agent pipeline in which the professor and TAs communicate with each other. The first step is the user's syllabus is sent to the professor agent. The professor would then send the different chunks of text to each group of TA agents. After analyzing the text, each TA group will summarize the chunk to the professor. The professor then decides how to format the test and tells each group of TAs to make exam problems about given topics. After all the different exam problems are created, Rajiv compiles them to create the final product for the user.

The back end is built on top of FastAPI and deployed through Google Cloud.

The front end is built on top of Next.JS and deployed through Vercel and GoDaddy.

Challenges

One issue we encountered was converting a PDF document (syllabus) to a text file. We struggled to find an open-source code that performs this task both on the server side and in JavaScript.

Another challenge we faced was fine-tuning our model. We used only a small sample size of data because fine-tuning it with large amounts of data is costly. The limited tuning produced lackluster results.

Learnings

We learned how to develop a multi-agent model with different models and functions. This was also our first time fine-tuning a GPT model.

What's next for RajivAI

We plan on introducing more functions to Rajiv and TAs, including but not limited to math, coding, and writing APIs. We also want to further fine-tune our TA models to produce accurate answers.