HOP - Homework Optimization Pal

• 

  Idle state

• 

  Analyzing state

• 

  Carrot pointer

• 

  Error display

• 

  Boundary boxes

• 

  Multi-mistake identification

Inspiration

Artificial intelligence has became an inseparable part of modern education. While some use it to cheat through homework, it is undeniably a great tool to check your work and find problems in your solutions. However, for a lot of us physical-notes enthusiasts, taking a photo to send to LLMs is not only a lot of work, but it's also often hard to find exactly where the error is base on the response of the AI models.

What it does

HOP is a study companion robot that lies on your desk and is interfaced through the front display. With the tap of the screen, HOP will identify all mistakes made in your calculations and physically point out the mistake using its carrot pointer with a short reasoning on why that line is a mistake.

How we built it

The HOP bot itself is being controlled by a ESP32 controlling two gearbox motors as well as one 2.8 inch display. It communicates with the Raspberry pi through serial with a physical USB connection and can send the capture command. The computer/raspberry pi handles image capture and interfaces with the backend, converting the API call results into compact serial protocol (move from a to b). The backend is a Fast API server which accepts API calls for image analysis via gemini and formatting everything into the proper json format.

Challenges we ran into

Our original plan was to use standard wheels on the gantry with rails to contain it. However, we quickly realized that we were losing far too much efficiency to friction and had inconsistent positioning. To overcome this we converted the wheels and track to both be geared allowing for accurate and efficient movement. Another similar case where we had to change our entire plan is when we realize that we don't have enough space in the bot to fit 2 ESP32, forcing us to re-wire all the pins for two motors and the display onto one ESP32. We even had to use some of the boot pins, meaning we have to hold the boot buttons when uploading code to ESP32 for some of our code.

Accomplishments that we're proud of

-calibrating stepper motors to travel to an arbitrary coordinate through pixel counts. -integration of two stepper motors and display onto a single ESP32-WROOM by soldering multiple female to female to the common ground (since there's not enough pins for every component) -Geared wheel and track system for precise and simplified movement through laser cut tracks

What we learned

We learned how to CAD quickly, use a laser cutter, soldering components, interfacing a computer with a microcontroller through serial, calibrating motors, using Gemini's native 2D bounding box and prompt engineering.

What's next for HOP

HOP has the potential to go completely wireless and trackless. With wheels that has proper grip on the table, the track can be removed, greatly reducing the overall size of the product. Aside from that, as our fast API is already on a deployment server and both our PI and ESP 32 has the ability to connect to the web, they would be able to communicate with each other without any physical wire.

Built With

• esp32
• fast-api
• gemini-api
• vercel