Diagram-Based Test Question Generator

Uses a dataset of diagram-based test questions to tune an LLM into generating similar questions based on a user-input diagram

Comment

Inspiration

I initially was unsure what my project would be, so I began searching online for datasets that could be used to contextualize an LLM for a specific task. I explored ideas like investment recommendations and document compliance checking, but I was unable to find sufficient data to provide to the model. Eventually, I found a dataset called "The Cauldron" on Hugging Face, which consists of a variety of diagram-based multiple choice exam questions. I got the idea for this project from the types of questions in the database, as well as how they used the information from the diagrams.

What it does

The program takes a file input from a user: an image diagram that illustrates some arbitrary concept (similar to those in the dataset). The image and dataset are then embedded in a detailed query to the Claude 3.5 Sonnet LLM that specifies the task, exact output format, and further necessary details. The model then generates a question, in the style of those from the dataset, that aligns with the diagram input by the user. The response consists of a question, four answer choices, and a detailed explanation of the correct answer.

How I built it

The project was built in Google Colaboratory using Python. The dataset was imported from Hugging Face using the "datasets" library, while all of the model and data input was handled with LlamaIndex Open Source. An Anthropic API key was used with the Claude 3.5 Sonnet model.

Challenges I ran into

  • Converting the data into a usable format, since it originally came in a parquet file
  • Finding a suitable model that could give me the desired result (also tried Gemini)

Accomplishments that I'm proud of

  • Creating a detailed prompt that gave me a result in the desired format
  • Learning how to use multiple new technologies from scratch

What I learned

  • How to implement LlamaIndex and the Anthropic API
  • How to engineer a prompt to create specific output

What's next for Diagram-Based Test Question Generator

I was unable to build a frontend for the application due to time constraints and other failed project ideas, so that would likely be the next step.

Dataset Link: link

Built With

  • anthropic
  • colaboratory
  • huggingface
  • llamaindex
  • python
Share this project:

Updates