Math Sandbox

Inspiration

Our inspiration for this project was Dynamicland, a concept where computing interaction is highly physical. Many videos showcase a real-life interface to computing which we thought was an excellent concept to demonstrate at a hackathon.

What it does

Math Sandbox is an interactive math exploration tool for exploring functions. Write your equation on a piece of paper and see it graphed on your sandbox! Any paper placed on the workspace is transformed into a piece of dynamic media with a combination of a projector and camera.

How we built it

Math Sandbox is powered by OpenCV, a computer vision library which helps us align projector output with camera input. Additionally, we use Google Gemini AI to parse handwritten equations into SymPy. SymPy works with matplotlib to plot the equations.

Challenges we ran into

Graphing equations in real time is slow and hard. It's easy to accidentally make expensive blocking logic run every frame. We had to work hard to optimize it enough to reach a proof of concept.

Accomplishments we're proud of

This was a pretty ambitious idea, so we were happy to pull it off in a shorter hackathon. One of the parts we enjoy the most is the Add card capability, especially its ability to render math formatting.

What we learned

One of our struggles was finding a good prompt for Gemini to output equations in the format we needed. Gemini liked to output Python code even though we told it to strictly output SymPy syntax. With a little bit of extra emphasis and persuasion, we were able to circumvent this difficulty. Another issue arose from the fact that for malformed inputs, Gemini would simply return our example output. We worked around this issue by asking it to explicitly return a sentinel string if it was not confident in its output.

What's next for Math Sandbox

We would love to iterate on stability and performance. Additionally, there are some user experience improvements to be made, such as better indicating the reason for failure to parse functions and adding animations that better explain how the cards interact with each other.