We came to this hackathon with a new set of skills and experience that we wanted to put to the test. We saw many exciting challenges to try and we decided to go for a couple of them that attracted us technologically speaking. We wanted to try out Typeform, a platform that we've seen HackUPC use time and time again and then we also went for the Floorfy challenge because it gave us a prompt to delve into the magic world of deep learning.
What it does
What we build is a tour recommendation system. After answering some queries the user will be presented with a set of tours and it's rooms that we recommend. The system also has the ability to display images with the points of interest outlined.
How we built it
First the data was preprocessed using a ResNet model trained on the CocoStuff dataset. Using pytorch we ran the object segmentation pipeline and saved the results. This results were then used to populate an Elasticsearch database ready to be queried. We have one backend running with Go and a front end using Typeform plus a microservice that has the Elasticsearch database with all the preprocessed data to query. The first two communicate with the micro service and it returns the recommended information. It is also capable of serving the images of the rooms with and without outlining the points of interest found.
Challenges we ran into
The fact that we tried to use quite a few different technologies proved a difficulty in terms of amount of stuff to research that we didn't know about. Doing stuff with Deep Learning with such a short time (and in the case of our data engineer, a computer without a GPU) was a laborious task so we had to decide wisely and be very sure that when we run a 12 hour processing pipeline it would be the correct one.
Accomplishments that we're proud of
Having made it this far is for sure a great feat! Sleep deprivation and tired eyes were a big rival throughout this adventure. We are proud that we managed to get an initial prototype with some functionalities working given the fact that the scope of the idea we had in mind was very large.
Also, we were all a bit rusty with front end and it was a bit hard to remember stuff all over again xD
What we learned
We learned about Typeform, which we hadn't used before and we got some more insights and practice on what you have to do when you use Deep Learning for "production" stuff instead of purely research (efficiency is key! we don't have a big cluster of gpus now!).
Also we learnt more about teamwork, we are a team who had never worked together before and we did quite well!
What's next for Tourify
Who knows? If someone funds a cluster and helps us out with front end we'll keep going with this heheheh.