Bruno

We want to opt in for the following MLH prizes:

Best Overall Hack
Best Use of AI in Education by Schmidt Futures

Background

As both students in and TAs in the Brown CS department, we have experienced firsthand the issues that students and TAs face in office hours. In many cases, students struggle to receive the help they need to learn, and TAs struggle to keep up with meeting the needs of so many students.

To further explore this problem, we surveyed our classmates and pinpointed the following issues:

TAs are overworked and answer redundant questions.
Students need more help than current solutions (EdStem, Office Hours)
With seemingly nowhere else to go, students turn to other resources like ChatGPT for answers.

Students already use ChatGPT for both debugging and conceptual questions. 79% of students report using ChatGPT before going to hours. 50% of students report using ChatGPT before using EdStem.

2/3 of TAs report getting asked the same questions frequently.

With hours, students report long waits to be seen, feeling pressured to go quickly, confusing or incomplete answers, and follow up issues after leaving.

What it does

We present Bruno AI, the first education-focused LLM that can adapt to specific course environments. Unlike competing AI products, Bruno AI helps students with conceptual, debugging, and logistic questions, informed by relevant coursework data including syllabi, assignment details and code, and EdStem posts.

How we built it

The front end is built using Streamlit. This integrates into our backend, where we use retrieval augmented generation (RAG) to query relevant coursework data.

To enforce Bruno AI to respond in a pedagogical and constructive way, we employ few-shot learning and chain-of-thought reasoning with carefully hand-crafted examples, techniques proven to improve accuracy. Guided by these methods, Bruno AI encourages students to engage with material by prompting students to explain their thought process rather than simply providing an answer.

To allow Bruno AI to provide course-specific feedback, we retrieve relevant snippets of the assignment spec, syllabus, code, and even similar EdStem questions to pass in as context when responding to a student's questions. To perform fast retrieval, we break apart large documents into smaller subcomponents, using cosine similarity on the embeddings of these components and the student's query to find relevant content.

This information is sent with carefully crafted examples and pedagogical instructions to GPT4. Finally, before we display the response back to the user, we the query and response for malicious intent to ensure that the LLM does not output misguided responses.