explAInotes

Inspiration

You've probably found yourself sitting in class feeling in a daze as the professor keeps teaching but you are not understanding anything. You have no idea what is going on, but there is no time to ask questions and you keep falling further behind.

What it does

That's why we made explAInotes, a notetaking web application to answer all your questions and clarify confusing concepts in real time while you are taking notes in lecture. Using stylus integration, you can write notes naturally. If you end a sentence with a question mark (?), the AI sidebar directly answers it. If you write an expression followed by an equals sign (=), the AI solves it. Otherwise, it simply defines the core concepts you are writing about. It turns passive note-taking into an active, real-time feedback loop.

How we built it

We built a custom web interface optimized for iPad Safari using HTML5, CSS, and JavaScript. We used a element to capture smooth stylus strokes, and we implemented a custom debounce timer that listens for pauses in writing. When a pause is detected, the frontend calculates a precise bounding box only around newly drawn ink and converts it into a Base64 image. We also built our backend using a Python server powered by FastAPI and Uvicorn which receives the image and passes it into the Gemini 1.5 Flash multimodal model. The LLM reads the handwriting and processes the logic, and sends its response in the dynamic sidebar.

Challenges we ran into

We ran into challenges setting an optimal debounce timer since setting it too short would send writing too quickly to the API and we would quickly reach the API call quota. However, setting it too long would reduce the live interactivity of the application. We tested thoroughly to find the optimal timing of 4 seconds.
Also, we wanted the AI to only look at the most recently written notes, which we solved by segmenting the notes by tracking the min/max x and y coordinates of the pen strokes to digitally crop the canvas.

Accomplishments that we're proud of

We are proud of being able to implement responsive notetaking and acheiving a native-feeling drawing experience inside a mobile browser. We successfully used a combination of visual data extraction and AI reasoning to parse the user's input. We are also proud that we had a very applicable and useful project idea that solves a very relatable and real problem. Best of all, this was our first time making an application that integrated an LLM.

What's next for explAInotes

Our next step is to implement the camera feature to take pictures of lecture slides or whiteboards and automatically convert the text into notes and seamlessly incorporate diagrams. We also want to implement user accounts for saving note history and cloud syncing across devices

Built With

• canvas-api
• css3
• fastapi
• gemini-1.5-flash-api
• git
• github
• hmtl5
• javascript
• python
• uvicorn