CalculusCat

Inspiration

On average, math education is—to put it lightly—incredibly poor. And to make matters worse, good quality math education is only accessible to a select few. When someone is faced with a homework question that they don't know how to answer, they usually just turn to the internet. While online calculators can get you a correct answer, they don't help you actually learn the material while you do so; showing their steps can only do so much.

Learning is a much more active process, something that is best done with the help of a good teacher or tutor. But, of course, quality tutoring is completely inaccessible to most. Our goal? We want to re-build how we do math help online, making high quality education free and accessible to all.

What is CalculusCat?

CalculusCat is a two part process. The core is a math tutor, who takes the form of a cat: Sir Isaac Meowton. Meowton is built on a large language model, which we specifically engineered and fine-tuned to be accurate and helpful for guiding a student towards solutions. We do so by adopting an "active questioning" method, where learning is best done not by seeing the solution, but by engaging in conversation with the problem. Meowton is specifically trained to not provide the answer to a problem, but instead to determine gaps in your knowledge and help you fill them through active questioning.

In addition to Meowton, we wanted to take advantage of some of the great—but, sometimes overly general—resources available on the internet. We hand curated a couple quality sources, scraped the data off of them, and used transfomer-based text summarization and semantic search to send these resources to you while you're interacting with Meowton. This way, you can get a better grasp of the underlying concepts in the problem your solving, through resources specifically designed for quality teaching.

How We Made It

We built the web app using Vercel, interfacing with the OpenAI API to access a GPT-3 model. We first fine-tuned this model on auto-generated arithmetic data, as well as auto-generated derivatives, in order to improve the models basic math capabilities. We then focused on prompt engineering, which involved writing a "scratchpad" as well as few-shot prompting the model before it was used. Scratchpads notably achieve higher accuracy than pure few-shot prompting, and our scratchpad allowed us to force CalculusCat to never directly provide the answer, instead only asking questions and providing guidance.

For the internet resource integration, we first chose a couple sources whom we know are useful for math tutoring (Khan Academy and 3Blue1Brown). Then, we scraped video transcripts from these two sources, compiling them into a large dataset. We used a pre-trained T5 model to summarize each of these transcripts, giving us shorter versions of each document; this prevented "extraneous" material (i.e. tangents) from confusing the semantic search. We then used a bidirectional transformer model to do semantic search based on sentence embeddings, taking CalculusCat's output as the query and using the entire body of transcripts as our corpus. Once we did this, we used a simple weighted average to score each summary based on how relevant each sentence was to the query, giving us a final ranking of videos. We then pass this video to the model for a recommendation.

Learning

This project was a huge undertaking for us, since only a couple of us have much experience with programming, and none of us have ever built a web app end-to-end like this. None of us have used Vercel in the past, so we had to learn how to do that over the course of the hackathon, and only one of us has experience working with APIs. A lot of our difficulty was purely just technical integration, so we each worked on the areas we had the most experience with, which made it especially exciting to see it all come together in the end.

Future Direction

There are a lot of places we want to take CalculusCat. One of the biggest is improving CalculusCat's math capabilities; currently, it's built on the davinci model of GPT-3, which is capable of doing math but is not infallible. In a way, CalculusCat sidesteps this issue by never directly attempting to solve a problem, which means all it needs to do is determine whether or not the user is correct. It does this with relatively high accuracy, but we'd still like to ensure that the backend model is really truly accurate in doing math.

Another big step is expanding CalculusCat's capability to beyond calculus, which we've currently restricted it to. This isn't too complex to execute, but it's definitely an important step in CalculusCat's journey to become a truly high-quality generalized tutor.

We'd also like to include more integrations with other math tools, like WolframAlpha, Desmos Graphing Calculator, and GeoGebra 3D. We'd also like to integrate a widget which would allow users to typeset their questions in LaTeX using MathQuill.

Built With

• html
• javascript
• openai
• python
• typescript
• vercel