Howl, the Holographic Owl

Object detection for 360° images is currently state-of-the-art so its models are still academic. With Howl, we exploit the accuracy of the best models to recognize and detect these as normal images.

Comment

Inspiration

I am a master's student in Computer Graphics and Virtual reality at UPC with experience in Data Engineering. But most importantly, I am host in Airbnb! That is why I know how important it is to have the best pictures to increase your chances to receive an offer.

When I saw the opportunity given by Floorfy to use their images dataset, I didn't have to think twice to focus on this challenge. I am very passionate about my career and I could not be happier to do it if I can also include my side project experience in Airbnb to present an interesting idea.

What it does

Howl (the Holographic Owl) takes full advantage of the best object detection models that the market provides. In order to do so, we use computer vision to convert the equirrectangular images provided into perspective images using skybox 3D method as the main idea.

skybox method

Once we have the images in a conventional format, we process them in deep learning models to obtain:

  • Object Detection to know which objects are in the image and its position if possible.
  • Room Type Classification to determine if we are in a bathroom, bedroom, outdoors, etc.

Object and scene detection

Once the images are processed, we do reverse engineering to get the 360° images again, but with the position of the detected objects as well.

How we built it

First, I spent the day downloading the dataset. Then, I made sooooo many tests with the data in order to figure out how we can exploit it using existing resources. I realized the 360° image format wasn't the best starting point so I tried different ways to build a workaround.

That's how the concept of an Owl came to my mind. A animal with good vision that can even turn 180° to never miss a spot, but never watching all at the same time. I made some samples of how an Owl will see using python opencv and process them using AWS Image Rekognition. The results were far better than using equirrectangular images.

Then I retroproject the conventional images using python opencv to get the original 360° images.

Challenges we ran into

The timezone become a huge disadvantage when finding people to team up :c.

Time flyes in the hackathon, I would have like to use Pytorch to recognize the images by myself, but I could not manage the time to do it.

Accomplishments that we're proud of

Proud to find the workaround to this current problem until models to detect objects in 360° images become better (Actually, it is a great idea for a master's thesis, I hope a company can be interested in it to patronize me...)

What we learned

  • Connect the dots is such a powerful tool in life to apply everything you have learned to generate more innovative ideas.
  • Realize how much there is still to be discovered in the field of images, and be glad to be part of it.
  • Take a shower if feeling stuck.

What's next for Howl

I would like to exploit more the data given by Floorfy and see if Howl can check classified scenes and recommend changes if necessary. Another thing I would like to do is to analize the llumination of the scenes and compare with data if it cause a huge impact to interested users.

Share this project:

Updates