eduBot.io

For our project, we are aiming to utilize the Gemma models provided to determine how well they could provide educational and informative answers to different science questions along different topics and educational levels. The dataset that we used comes from Kaggle, and consists of 50,000 rows of data with the following columns: id, subject (chemistry, physics, biology), grade_level (elementary, middle, high), topic (80 different science topics), student_input (the prompt for the tutor; in our case, the LLM), tutor_response (an response provided by a knowledgeable tutor providing age-appropriate explanations), data_type (conversation, homework_help, direct_qa), and intent (educational_help, homework_assistance, knowledge_query). Our goal is to measure whether the model can provide accurate, adaptive, and relevant information in an efficient manner and with efficient usage of resources. For our project, we hope to measure and evaluate the following: accuracy (how accurate in terms of cosine similarity is the model to real answers provided by tutors), adaptability (how adaptive is the model to the depth of a response, such as accounting for student level and topic), and stubbornness (how much the model changes its responses). The key metric for our project is our adaptability metric, targeting how our model can adapt and cater to the needs of a student based on the subject and grade level information. To measure adaptability, we are going to prompt the model with the following information in three different intervals: the individual student_input, the student_input and subject, and lastly the student_input, subject, and grade_level, investigating how our model performs as a baseline and with additional context. In addition to adaptability, we plan on expanding our analysis with stubbornness, which measures how our model responses change at each pass. To measure our model over different languages, we’ll be testing performance with Spanish to see whether the model performs differently. Can our model adapt well to help provide accurate, adaptive, and reliable tutor answers in an efficient and effective way to help educate students?