AutoVision

• GIF

  GIF of AutoVision scanning a breadboard, on three axes

• GIF

Inspiration

We were inspired by an already implemented solution in automated factory lines. Many factories use cameras with computer vision from many angles to asses whether a package is viable to ship. We thought, why not give an AI the same capability to give itself context on prompts involving physical objects?

What it does

AutoVision uses a 3 axis robotic arm with a camera to give Gemini the power to get multiple angles of a physical product. Gemini is able to send commands like "top view", "side view", "back view", giving it the power to control its context and get the information it needs to improve the response.

How we built it

The arm works using 3 servos to control an arm that can go to the side and above, along with a stepper motor that has the capability to rotate the platform. The file is written in python, which runs the UI and sends API calls to Gemini. When context is needed, it sends commands to the arduino to get multiple angled photos, which then use the camera to add the Gemini's context.

Challenges we ran into

We ran into multiple challenges throughout the build:

• Two of our motors got fried, which we had to replace constantly (re-glue!)
• Our lack of limit switches meant in testing, the arm would often break itself
• Getting our camera connected to Gemini, and timing the arm completing its process and the camera the images was difficult
• Programming the controls and predefined positions of the arm was also difficult

Accomplishments that we're proud of

• We were able to successfully get the 3 axis arm working
• The gemini api is successfully able to add to its own context via the arm
• The controls allow multiple angles
• The UI is clean and easy to understand

What we learned

• We learned lots of hardware and software skills, working with drivers, arduino, and connecting to a web UI via python
• We learned how stepper motors and servos work, how to use drivers and connect the electronics
• We learned about how to use python to connect UI (web interface), and physical servos, which was really awsome.

What's next for AutoVision

• We plan to make a more stable arm, more angles, and ship demos to see possible use cases for engineering students or other useres
• We plan to run serveys of how often people use photo abilities with AI for context, to see which demographics would use it the most

Built With

• arduino
• c++
• embedded
• motors
• python
• streamlit
• uart