Our inspiration for the project was based on augmented reality glasses, where we imagined ourselves looking at stores and automatically getting all the relevant information from them. Since we didn't have access to augmented reality glasses, we decided to use our phones to accomplish the same task.
What it does
NearbyNow is a service that can scan storefronts such as shops, restaurants, grocery stores, and so on, using a live camera feed. The service uses two distinct methods to scan for storefronts.
The first method uses a custom-trained MobileNetV2 Deep Neural Network to predict a storefront from a trained dataset. Using a DNN allows us to train more specific elements to detect in the future such as an open/closed sign, the amount of traffic inside the store, and so on.
If the DNN yields a low prediction threshold, we use our second detection method, which is composed of two sub-steps.
Firstly, we send some of the frames of the live camera feed to be processed by our Optical Character Recognition algorithm. If a shop is detected, we go on to step two.
Secondly, we utilize the Google Maps API to cleverly match a store with our results. based on the user's location we compare the shops nearby and output a relevant result. Once this is done, we display our findings to the front end.
Link to diagram: diagram
How we built it
- Brainstormed the idea
- Split the tasks into front-end and back-end sections
- Created the dataset for the DNN
- Designed and custom-trained the DNN
- Created HTTP requests for the APIs
- Integrated the OCR
- Parsed the data received from the APIs (OCR, Google Maps, Geoapify)
- Integrated the DNN and OCR into the front-end
- Created an API call to compare nearby stores using geolocation based on a given radius
- Compared the OCR prediction using nearby stores
- Queried the data needed for the given store
- Displayed the information retrieved on the front-end
Challenges we ran into
- Creating the dataset for custom analysis proved to be difficult and time-consuming, as finding relevant and good photographs was a challenge
- Tuning the DNN to match the required accuracy results needed for our analyzer to output the correct information
- Make sure our DNN is lightweight to run on a mobile device
- Making sure to detect the right store when the video was noisy or had multiple other texts
- Communicating with all the APIs and gathering the correct information
- APIs having strict limits on how many times we could call them and the output limits
- Displaying the relevant information when not always available (ex: website or customer rating)
Accomplishments that we're proud of
- We were able to design, train and run a DNN network with good accuracies to predict stores CNN results graph
- Created a beautiful front end to display our results
- Achieved OCR readings from a live camera feed
- Integrated multiple APIs and combined them together to get an accurate and satisfactory result
- Used GoogleMaps API to find the nearest stores using geolocation within a radius in order to find the correct store and fetch the necessary data
- Used String pattern matching to detect anomalies and invalid OCR readings
What we learned
Creating NearbyNow, we learned how powerful APIs can be, as well as the complexity of the task. Pattern Recognition using CNNs, live OCR readings, front-end development and creating a dataset can all be separated into individual problems that we could spend countless days improving. However, we are very proud to have accomplished what we did in 24 hours.
What's next for NearbyNow
A lot of improvements and additions can be made to gain more accurate results and have more relevant information displayed when scanning a storefront. Eventually, we would like to add more detailed reviews of businesses, be able to detect traffic in the area and be able to recognize and display important historic buildings and monuments.