BarPlaybook

Inspiration

When I was studying for the California bar exam, there were a lot of options for how to study for the multiple choice section. But there were very few ways to get constructive feedback on the essay portion. California bar exam essays are extremely complex and nuanced, and the grading criteria spans several different factors, including legal analysis skills, knowledge and understanding of the law, application of law, and proficiency in using and applying legal principles, and more. As such, they are difficult to grade and provide feedback on. Right now, the only way to get useful help on your bar exam practice essays is to utilize a human grader. This approach was too expensive and time-consuming for me, and ultimately I didn't practice essays as much as I'd wanted to.

What it does

BarPlaybook tries to address this gap in study prep for the bar exam. Incorporating formal guidelines from the State Bar of California and other resources, BarPlaybook enables highly detailed, dynamic prompting to OpenAI's gpt-4 to review a user's essay instantly, compare it to a model answer, provide an accurate grade, and offer detailed and constructive feedback for how to the user can improve their approach. It also saves user inputs, and will soon be able to consolidate all their data (essays, AI feedback, scores, etc.) to provide realtime, high-level insights into how the user can improve their approach to the CA bar exam essay portion.

How we built it

BarPlaybook is a cloud-based web application that uses Supabase, Vercel, React, OpenAi, and Langchain to receive user essays and process them to provide the user an abundance of useful information for their studying needs. I've structured approx 100 previous bar exam essay questions for users to choose from. Then they start a timed essay session of 1-hour to practice their exam. When they submit their answer, it is fed into a Langchain function that runs a series of prompts against the user's essay. First, OpenAI is prompted to review a model answer of the essay question and create a rubric for how a 'hypothetical' applicant would answer a question based off the model answer. Then OpenAI takes that first output and compares it to the user's input. From this, OpenAI can make a fairly solid assessment of how well the user did. This assessment is done twice to ensure nothing has been missed. Then a further prompt establishes a score for the user. And at the end, a consolidated set of feedback is provided to the user on the frontend.

Challenges we ran into

One of the most difficult challenges was getting the AI prompts to output a realistic grade for the essays. At the beginning, the AI gave grades that were too high compared to what they got in real life. I was able to test the accuracy of the model based on the fact that I have a large repository of real-life essays (and the grades they received) from past bar exams. Continuously iterating on the prompts and testing the grades allowed me to finally start getting accurate scores.

The limited context window continues to be a challenge.

Accomplishments that we're proud of

The most significant accomplishments are:

• getting the AI to produce scores consistent with what the essays received in real-life;
• getting the AI to produce sufficiently detailed and constructive feedback that a law student would find useful in their studying;
• working around the context window limitations to give the AI enough background info to know 1: what a good answer to a question is, 2: why it's a good answer (per Cal Bar parameters), 3: applying those "why" answers to the user's essay so that it can give useful feedback to the user.

What we learned

There are many different approaches to getting an acceptable result from the AI. Giving the AI proper context is absolutely essential. Iterative prompting, and building into a final answer using progressively more complex AI inputs-outputs allows for an infinitely more interesting end result than simply using one or two calls.

What's next for BarPlaybook

Testing with beta users!

Built With

• langchain
• nextjs
• openai
• react
• supabase