DL - Vathia Mathisi

Inspiration

When given a real life image or video, we might wonder where is the picture locate. With a database of images that contains accurate coordinates constructed, the problem now changes to how can we match the images in our database with the query images.

What it does

The model is able to match the given image to the top 3 images within our database, even with differences such as time, angle and background.

How we built it

We first use a pretrained vgg16 network as the cnn backbone and cut the network at the last conv and pooling layer. The output feature map will be viewed as a K*D-dimensional descriptors of the image and feed into the netvlad layer. The netvlad layer then do soft assignment on each descriptors based on the cluster center which is done by using a conv layer.

The training dataset we use is pittsburgh 30k and use triplet margin loss and sgd for training

Challenges we ran into

Our members are of different backgrounds and skill levels, so it was challenging at first to bridge the gaps in understanding, but we managed to make it work. Most of us are also busy with schoolwork and other commitments, so it took a lot of coordination to get things done within the time constraints, but we are glad we made it work.

Accomplishments that we're proud of

We are very proud of how good the model is performing (e.g. how well it works with the images provided). Our group also consists of people from different backgrounds, schools, majors, etc and I think the fact that we still managed to come together and get something done is incredible.

What we learned

How it feels like to work in a team (specifically for a hackathon). That was definitely an enriching experience for all of us. Additionally, we learnt alot from each other with regards to the technically aspects as well (neural networks, python, deep learning, etc)