Compass Logo
Architecture
KNN Close Friend Algorithmn
KNN User Graph
User Powered Recommendations
Event Modal
Restaurant Recommendations
Viewing Real Time Events
Hack the North Modal
Data Pipeline

3rd Place Facebook API Prize

PLEASE WATCH THE DEMO VIDEO IN THE HEADING OF THIS DEVPOST

compass Inspiration

Through our travel experiences, we realized that planning a trip catered to our preferences is very difficult. Often times, top tourist destinations are over hyped and not suited towards our interests. One way we sought to fill this need in the market is through the creation of wandAR. Our inspiration stemmed from our fascination in the growing fields of AR and virtual worlds, from full-body tracking to 3D-visualization. We felt that the prime way we could display our interest in this technology would be in harnessing its ability to connect with friends and receive interactive event information. This is what led us to create wandAR, a technology that allows users to view live events happening around them in AR. This app has two primary modes, suggestion and explore. In suggestion mode, users receive match suggestions based on their own and their friends interests. In explore mode, the user has the ability to self guide their adventure and explore the city in AR. Local events are served with live updates. We envision that wandAR could revolutionize the travel industry.

The scope of our hackathon project includes:

Match Algorithm (K-nearest neighbors algorithm): We began this project by mapping out a user's best friends. From the Facebook graph API we received each user and their mutual friends. We look through each user's page likes and event data and pull the tags for each event. We aggregate this data and create a basic tensor composed of the user's price range and categories they have liked. The basic tensor consists of each price range ($, $$, $$$, and $$$$) and the union of all categories and tags associated with a user. These tensors are populated and through iterating over all the users, who are now described under a similar tensor. Then, the KNN algorithm is run on the non-normalized tensors; the result of the algorithm will rank each of the tensors relative to each other in an n-dimensional space, where n is 4 plus the size of the union of the tags. We then develop an in-place binary search tree modeled through an array and efficiently get the K "closest" friends in O(K) time, while building the tree itself in O(U^2) time, where U represents the number of users in the data frame. After that, take the complement of the set of the intersection of the user's data with all the data (to remove the user data) and use the absolute extrema to calculate relative percentages of friendship. What we get at the end is an ordered list of data that represents a best friend list, and these values are used in a weighted average algorithm to select key events to show to the user which we believe they would most enjoy.

Facebook Integration (graph API): Facebook was our primary source of data in this hack and was fundamental in the success of our application. We used the Facebook Graph API to get all the information we used to compute our best friend algorithm and the information we displayed in our AR view. The largest issue we encountered in our endeavor was that Facebook's API had recently limited its functionality and did not allow us to access our friend's data even if it were public. In order to overcome this, we manually found and added test users to our app and got them to authorize permissions which would allow us to receive data on which events they were going to and which pages they like.

Another issue with data from Facebook was that some of the pages and events we tried to use didn't have proper addresses associated with them, so we had to resort to using Google Places API to reverse search the venue names and get associated coordinates.

AR View ( AR-kit, Swift ) : The compass iOS app renders interactive hover planes which annotate the world around us. We have the ability to look at planes with information about events and click on them in order to receive more information. We also have the ability to point our phone at local restaurants and view comments about the establishment left by friends on Facebook. All the modals update live and allow the user to immerse themselves into a more informative real world.

Technically, this was done using an advanced formula which combined slower updating, imprecise GPS and compass data with the rapidly updating, precise data from the accelerometer to produce high quality location data for tracking of the user in the scene. This allows us to move the annotations as the user walks (accelerometer at the microscale, GPS at the larger scale) and as the user looks around (accelerometer + compass data). Another challenge was mapping the longitude and latitude coordinates to actual AR scene coordinates to place the annotations at the proper virtual location, which involved a number of 3D heaversine-based calculations.

Firebase (Cloud Functions, Firestore): We used Google Firebase to bolster authentication for our web platform. We used Facebook login as it ties in seamlessly into our application and stored userID and username in the Firestore database. The reason for relying on Firebase is due to its proficiency for dealing with real time operations, as it ties in very tightly with cloud functions. We used cloud functions to host the majority of our backend, including the actual retrieval of the Facebook posts, parsing of the posts, ranking and final sending of the responses to the mobile client.

Architecture:

We used Firebase Cloud Functions as an entry point for our application. These are functions that are written in NodeJS and hosted on Firebase. They directly access the Facebook Graph API, create an authentication gateway using OAuth to connect to Facebook, and also a local Flask server hosted using Firebase.
The Flask server is connected to a machine learning cluster, where the KNNs and all recommendation engine calculations are performed. There's also a Jupyter notebook that can be viewed for a more modular approach to how we modeled the problem, and this was abstracted into a server to allow external use by the iOS application.
The iOS application directly interacts through the Authentication Gateway, and actively calls the Facebook Graph API to retrieve data and the Flask server to retrieve recommended suggestions based on best friends. The iOS app also implements ARKit and Core Location to show the hovering model above a location.