FlyVac

FlyVac is a, autonomous, non-destructive method of countering the invasive spotted lantern flies by using a vacuum on an SO101 robotic arm and the Nemotron Nano-Omni VLM to identify and suck up flies.

Comment

FlyVac

Inspiration

Our teammate's boyfriends dad lives in Hershey, Pennsylvania, and has to deal with thousands of spotted lantern flies devouring his grape vines on the daily. Every time he squishes fifty of them, hundreds more show up. FlyVac is an autonomous, non-destructive solution to deal with invasive insect populations, specifically lantern flies, that wreak havoc on farmers and backyard-gardeners alike.

What It Does

A stream of visual data from the camera is fed into the classifier, which identifies lantern flies. At a certain confidence level, it calculates the path needed for the arm to reach the lantern fly. After a path is calculated, instructions are sent to the robot arm, where they are interpreted and executed.

How We Built It

This was built using a combination of computer vision, VLM implementation, and embedded control. We first used video feed from an Intel RealSense D435i camera which was processed using OpenCV on Python. We then created a YOLO classifier and used the VLM to detect and confirm the presence and position of lantern flies. After this, the program calculates the 3D distance between the SO101 robot arm's end effector to the position of the lanternfly. Once calculations are complete, we use inverse kinematics to control the arm to "vacuum up" the bug.

Challenges

  • We ran into many issues with the Intel RealSense D435i while trying to properly stream data to our laptops. The camera requires USB-C 3.0 to fully function, while all but one of our laptops have USB-C 2.1, which costed us a lot of time debugging. The Python and OpenCV version mismatch between our computers and virtual environments also caused issues with getting the camera to work properly.

  • When training the Nemotron VLM to accurately detect lantern flies, we ran into many issues with the VLM detection. We did not fully understand the limitations of Nemotron at first, so we had to experiment a lot to get the classification functioning properly. We used a high contrast background and avoided reflective surfaces to reduce noise in the video feed, and had to simplify the complexity of the detection so that it would be functioning for our minimum viable product.

Accomplishments

  • For most of us, this was a good introductory experience into physical AI. We got to learn how to use the SO-101 and implementing a vision language model. We also learned a lot of limitations that we have to account for when it comes to hardware hacks.

What We Learned

  • Open Computer Vision Library for Python
  • LeRobot
  • WSL
  • Implementing VLM models
  • Calibrating RGB/Depth Cameras
  • Integrating a webpage using Spark, HTML and JavaScript as a dashboard.

What's Next for FlyVac

  • FlyVac: There are a lot of optimizations that we can do involving the entire system. For example, we still need to optimize the transformations between the calculated distance from the robot's end effector to the desired bug position. We can also expand the pipeline to adapt to multiple environments, which allows the VLM's capabilities to shine more.
  • Jacob: This is related to my master's thesis, where I have to work with vision language action models and the SO101. This is a nice introduction to it.

Built With

Share this project:

Updates

Submission history