Inspiration

As many local and small businesses lose money in sales during quarantine, many are looking to sell online. Figuring out how to best ship and package your merchandise is an important and expensive part of online business.

What it does

Paccelerate utilizes computer vision (OpenCV) and the Paccurate API to allow users to print out some QR codes and snap a quick picture for instant package measurement and packaging suggestions. This saves the user significant time, as it automatically makes rough measurements of all your objects for you, so all you need to do is enter their weight. You then get in return a simple diagram on how to fit your packages into one cost-effective shipment.

How we built it

The main aspect of this app is computer vision. A QR code is placed on the ground that has a special background tag with is the text "PaccuratePad:3:3:Back" or PaccuratePad:6:6:Back" with the numbers being the dimensions in inches, encoded as a QR code. We know the size of the QR code in real life and can, therefore, estimate how many pixels are in an inch at that depth in the image. Then we search for all the QR codes on top of objects. We know those QR code's dimensions as well since these QR codes are encoded to say "PaccuratePad:3:3:Front" or PaccuratePad:6:6:Front". Using these QR codes we can easily measure the length and width of any object in the image. We then superimpose a number on top of all the objects so that the user knows which object to refer to when inputting the weight.

Challenges we ran into

Finding an object's depth is the real issue with a 2D image. The user isn't providing us with multiple angles, we don't know the calibration of their camera, and we don't have an idea of any of the reference angles in the image. After hours of brainstorming, the solution we came up with to find image depth determines each QR code's actual size (in inches), its size in the image (in pixels), the focal length of the camera (in mm), and the digital zoom of the camera. To find the focal length and digital zoom of a camera, we use the command-line tool ExifTool, which gives us all the metadata captured from the phone's camera about the image and the camera. Then we can scan the QR code in the image to find its size in real life, and can use OpenCV to easily get image size. After all this data has been collected, we run

def depth_finder(object_width, real_width, width, focal_length, zoom):
        object_width = (zoom / focal_length) / width * object_width
        perceived = (math.pi / 180 * focal_length) * object_width
        object_distance = real_width / 2 / math.tan( perceived / 2 )
        return object_distance

which successfully returns the object's depth to about 1/4 inch accuracy!! Below is the JSON our computer vision returned for the example images in the slideshow attached:

{
        "image": "data:image/jpg;base64,/9j/4AAQSkZ...",
        "objects": [
        {
            "item_depth": 15.03,
            "item_height": 3.02,
            "item_number": 1,
            "item_width": 3.0
        },
        {
            "item_depth": 28.29,
            "item_height": 2.89,
            "item_number": 2,
            "item_width": 3.0
        }
    ]
}

Accomplishments that we're proud of

Getting OpenCV to properly detect and find the QR codes and objects was a huge struggle. The edges are very rough and we wanted this to be applicable anywhere with ease, so we couldn't just swap in a green screen as the background. We ended up sample pixel colors around the QR code to create an image mask that gets applied, removing that specific color range. This leaves that object black and makes it much easier to use edge detection on.

What we learned

OpenCV is an incredibly powerful, though complicated, tool. The things that can be accomplished with it are impressive, although given more time a ML approach to detect boxes may be a more robust idea.

What's next for Paccelerate

We hope to incorporate AR to measure the images more accurately from the client-side, eliminating the need for our API and therefore cutting an average of 1-2 seconds per image. We would also like to integrate our Paccurate API response with EchoAR to show the user a 3D AR model of exactly how their products would fit together to be shipped and so they could see just how much space it would take up.

+ 3 more
Share this project:

Updates