Toyota Innovation Challenge

• 

  Hole detection

• 

  Circle Detection

• 

  Filter application

• 

  Contour Detection

Background:

Toyota is developing a system that will automatically install round stickers over vehicle body holes to prevent damage. The automated system will apply the individual stickers over the designated holes. Our goal is to design an inspection system to:

1. Confirm the presence of all 7 stickers applied by the robot
2. Confirm the stickers fully cover the body holes
3. Confirm that the stickers are flush with the surface
4. The total inspection time for all 7 stickers should not exceed 15 seconds

First attempts:

We first attempted to write code to find the holes in static images. To do this, we used the houghCircleDetector function in OpenCV to display circles over the holes. After changing the sensitivity parameters of the function, our code managed to find some holes in the image, but either found too many holes or too few holes.

We then tried to use the detectShapes function to find the contours of the image. However, the image was too complex and had to be simplified by the pointDetector function, which adds a filter to the image. At this point, our program could find the holes in static images.

Live feed:

After testing our code on static images, we attempted to connect to a live feed camera. Using Python and Java, we processed the output from the live feed and implemented our previous processes on the frames. We first use a point detection filter to simplify the image and then use a shape detection filter to find the holes and stickers. At this point, the program could recognize shapes that are similar to circles.

Detect stickers:

We stored the coordinates of the detected circles in an array. If the circle is not symmetric (the length and width are not the same within the margin of error), it is classified as a failed sticker. This works with a live video and in 3 degrees freedom. We then calculate the average radius of all circles, excluding the failed stickers. Circles with a radius less than the average are classified as a hole.

Evaluation:

Our approach and scope changed throughout the challenge. We began by using filters and color space changes to identify the circles. We then switched to the Hough Circle Transform which worked adequately for single test images but could not be generalized without tuning its parameters. Our current solution leverages point and shape detection along with checking the size of the stickers to determine if we are looking at a hole, sticker, or partially covered hole. We are also attempting to solve the problem using YoloV5 and a Neural Network to recognize stickers. However, this requires a lot of training images. While our current solution works well, if we were to undertake this project again, we would work with a neural network from the start. It would serve as a more general solution and could be continually improved on by creating more layers and more training data.

Built With

• opencv
• python
• yolo