Formula-forge

A multimodal AI tutor: type, upload, or draw a math or science problem and get step-by-step solutions, hints, feedback on your own work, and practice problems. Learn the method, not just the answer.

Comment

Inspiration

Most AI homework tools focus on producing answers rather than helping students understand the underlying concepts. I wanted to build a tutor that teaches through guided hints, step-by-step explanations, independent answer checking, and interactive visualizations. Formula Forge was designed to let students learn naturally by typing a question, uploading a problem image, or drawing equations by hand.

What it does

Formula Forge is a multimodal AI learning copilot for mathematics, science, and engineering. It combines AI-powered problem solving, mathematical verification, interactive visual explanations, and personalized practice in one learning workspace.

  • Multimodal problem input: Type a question, upload a photo or screenshot, paste an image using Ctrl+V, or draw a problem on an interactive canvas.
  • Multiple learning modes:
    • Solve: Provides step-by-step explanations that describe the reasoning behind each step.
    • Hint Only: Offers progressive hints to guide students without immediately revealing the answer.
    • Check My Work: Reviews a student's attempt, identifies errors, and explains how to correct them.
  • Interactive visual learning: Generates supported mathematical plots, visual explanations, and diagrams. Students can adjust parameters and explore how changes affect the results.
  • Mathematical verification: Uses a custom expression evaluation and checking engine to independently validate supported calculations and provide verification status.
  • What-if simulations: Lets students explore alternative values and scenarios through AI-assisted predictions and interactive visualizations.
  • Personalized learning: Tracks quiz performance by topic, estimates mastery, identifies weaker areas, and recommends targeted practice.
  • Contextual AI tutoring: Supports follow-up questions, similar practice problems, quizzes, and explanations adapted to the selected subject and difficulty level.
  • Handwriting transcription: Converts drawn problems into editable text before solving, allowing students to correct misrecognized symbols and numbers.
  • Learning workspace and history: Organizes solutions, visualizations, verification results, and practice activities into dedicated sections. Supports saving problem history, copying solutions, and exporting content.

How I built it

  • Frontend: React, TypeScript, Vite, and Tailwind CSS, with a responsive interface, dark/light themes, reusable components, and an organized learning workspace.
  • Multimodal AI: Integrated Google's Gemini API for interpreting text and images, generating explanations, transcribing handwriting, and supporting different tutoring workflows through mode-specific prompts.
  • Backend: Built a Vercel serverless API that protects the Gemini API key, validates incoming requests, handles image payloads, and supports model fallback configuration.
  • Mathematical evaluation: Developed a custom expression parser and numerical evaluation utilities for supported mathematical operations, including function evaluation, derivatives, definite integrals, and root checks.
  • Interactive visualization: Implemented canvas-based function plots, adjustable parameters, shaded integral regions, and structured visualization outputs.
  • Drawing canvas: Built a custom stroke-based canvas with undo and transcription review. The drawing is cropped, scaled, and redrawn on a clean background before being sent for AI interpretation.
  • Adaptive learning: Implemented quiz-based topic tracking, mastery estimates, learning history, and targeted practice recommendations.
  • Input optimization and safety: Added client-side image compression, request validation, and Markdown/HTML sanitization to improve reliability and reduce unsafe rendering.

Challenges

  • Handwriting recognition: Improved the quality of handwritten input by preprocessing canvas strokes and introducing an editable transcription step.
  • Mathematical reliability: Added independent checks for supported calculations rather than relying entirely on AI-generated answers, while recognizing that numerical verification has limitations.
  • Canvas coordinate alignment: Fixed pointer-positioning issues caused by differences between the canvas's displayed dimensions and its internal drawing dimensions.
  • Model availability: Addressed retired Gemini models through configurable model selection and fallback handling.
  • Request-size limitations: Introduced image resizing and compression to reduce upload failures caused by large images.
  • Integrating multiple learning workflows: Connected solving, verification, visualization, quizzes, and follow-up questions into a consistent interface while managing loading states and API errors.

What I learned

Building Formula Forge strengthened my understanding of multimodal AI integration, prompt engineering, React and TypeScript architecture, serverless API security, mathematical expression evaluation, interactive data visualization, image preprocessing, and AI application reliability. I also learned the importance of separating AI-generated explanations from independently verified calculations and designing interfaces that encourage learning rather than answer copying.

What's next

  • More robust symbolic mathematics and verification for a wider range of expressions.
  • Improved localization of mistakes directly on uploaded handwritten solutions.
  • Real-time conversational voice tutoring.
  • Multimodal textbook and classroom-material understanding with retrieval-augmented generation.
  • More comprehensive automated evaluation, including transcription accuracy and answer correctness.
  • Expanded teacher and classroom workflows.

Live demo: https://formula-forge-phi.vercel.app
Source code: https://github.com/Santosh02411/Formula-Forge

Built With

  • adaptive-learning
  • ai-tutoring
  • computer-vision
  • gemini-api
  • generative-ai
  • git
  • html5
  • image-processing
  • interactive-data-visualization
  • javascript
  • mathematical-verification
  • multimodal-ai
  • numerical-analysis
  • prompt-engineering
  • react
  • responsive-web-design
  • rest-api
  • serverless-functions
  • speech-recognition
  • tailwind-css
  • text-to-speech
  • typescript
  • vercel
  • vite
Share this project:

Updates

posted an update —

Formula Forge is now live as a multimodal AI tutor for math, science and engineering. It teaches the method, not just the answer.

What's new

  • Three ways to enter a problem: type it, upload a photo or paste a screenshot, or draw it by hand.
  • Read my drawing: the AI transcribes handwriting into editable text, so you can fix any misread digit before solving.
  • Three learning modes: Solve for a step-by-step solution, Hint only for guided hints without the answer, and Check my work to find the first mistake in your own attempt.
  • Level and subject controls: Beginner, Standard or Advanced, plus Math, Physics, Chemistry, Biology, Engineering or Logic.
  • Follow-up questions and practice: ask why a step works, or get a similar problem to try with a self-check answer.
  • Export: copy a solution, download it as Markdown, or print it to PDF.

Under the hood

  • A custom drawing canvas with undo that re-renders your strokes clearly before sending them to the model.
  • A Vercel serverless function that keeps the API key private, validates inputs, and falls back to another Gemini model if one is unavailable.
  • Images are compressed in the browser so large photos don't fail.

What I fixed along the way Early versions misread handwriting and drew the line away from the cursor. The new canvas and the editable transcription step fixed both.

Next LaTeX math rendering, a history of past problems, and classroom features for teachers.

Try it: https://formula-forge-phi.vercel.app Code: https://github.com/Santosh02411/Formula-Forge

Log in or sign up for Devpost to join the conversation.

Submission history