Most states in the U.S. have issued some form of stay-at-home order as a means of preventing the spread of the novel coronavirus. Some states are starting to relax those orders and allow some types of businesses to reopen under certain conditions. Texas and Florida, for example, will allow retail stores, restaurants, movie theaters, museums, and libraries to reopen as long as they limit the number of people inside to 25% of their maximum capacity.
While this offers benefits to businesses and customers alike, it also creates new costs and risks. Businesses must now pull employees away from their regular duties to watch the doors and keep a careful count of how many customers have entered, and how many have left. Customers now run the risk of arriving at a store only to find that they need to wait in line before they can enter. Without proper social distancing, these lines create the exact risk of exposure to the coronavirus that all of the stay-at-home orders were designed to avoid in the first place.
Crowd Scout is essential to allowing customers to enjoy the social benefits of resuming normal daily activities, like shopping and entertainment, while still minimizing their risk of contracting COVID-19.
What it does
Crowd Scout can help our society and economy reopen safely by using technology to enhance the effectiveness of social distancing and reduce the risk of transmission of the coronavirus. It lets people be social again in a safe way by reducing the likelihood of overcrowding in public places (restaurants, bars, movie theaters, stores, etc.).
In particular, Crowd Scout is a mobile app designed to keep track of how many people are inside a specific area like a store, and shares that information with employees and customers alike.
During the current COVID-19 pandemic this will allow customers to see each store’s maximum customer capacity, and how close a store (restaurant, theater, etc.) is to reaching that capacity. This will save customers both time and possible coronavirus exposure by allowing them to find a store with the least number of other customers.
After the COVID-19 pandemic eventually ends, customers can still use the app as a time-saver to find which businesses have the fewest customers, especially during traditionally high-volume periods such as: holiday gift shopping; holiday grocery shopping; Friday and Saturday night theaters, restaurants, and bars; Sunday brunches, etc.
How I built it
Crowd Scout is a mobile app built using Flutter and Firebase. Flutter is the perfect tool for Crowd Scout because it allows for cross-compatibility between platforms and devices, which is essential for a tool designed to help everyone during a pandemic. It also promises better inter-connectivity with Google’s Cloud services, which is helpful, since our app is highly integrated with Google Maps, Firebase, and Google’s Cloud AutoML.
We used Firebase for our app’s database to store the numbers of people at each location, along with the information regarding max capacities. We used Google Maps for everything regarding the user’s location, along with autocompleting search to help users find the data they need, as well as the Google Places API to help track the user’s current location. We hope that implementing as many features of the Maps API as we could will help users feel comfortable within the app, and will allow them to fluidly and naturally access the data. Things like being able to select points of interest from the map allow users to navigate the data like they might in the real world, and increases the “fold” of the application to whatever a user can see on the map. We think that this will help people use the app to get data on locations they might not have thought about, like a local park, or coffee shop.
We also worked with the Google Cloud AutoML platform to try and create a model to better determine if the user was in a building or not. While we got the model trained, unfortunately it's accuracy was too low to use due to both the low quantity of training data, and the similarity between the user's position circle, and the blue point of interest markers already on the buildings.
Challenges I ran into
One challenge we ran into was in using the Google Cloud AutoML platform to check if a user is inside of a building. We were attempting to detect on the map screen if a user’s location was inside of a building on the map by looking at the user’s position in the satellite view. While we had limited success on a small scale, we didn’t have enough training data to make sure the algorithm was accurate for non-standard locations, like a park. We also had difficulty with detecting points of interest markers as a user. In the future, we’d like to revisit this to increase processing speed, and decrease overhead, while increasing accuracy.
It was also complicated get the place closest to the user's current position. There are APIs already built for this in the Android and iOS SDKs, but because we were using Flutter, we were not able to access those SDKs. In fact, we only had access to the HTTPS API, so we had to entirely write our own search and analysis algorithms in Firebase Functions to replicate this and ensure high quality data without slowing the app down for the user. The biggest challenges that came into play here were A) selecting the correct building to update, and B) not updating buildings that the user is not actually inside of.
With respect to part A, the original plan was to just use geocoding to get the address from the coordinate points. And this worked to a degree, we were able to turn every latitude and longitude point into an address. The catch was that when we then ran a text query to get the place associated with the address, we found that a lot of buildings physically sit on multiple addresses when the original point is geocoded, but the place information was only associated with a single address. This is the equivalent of tracking the number of customers in the garden section and the grocery section of the same store separately, which was problematic because capacity for all the sections of the same store are controlled at the entrance, so all that user data needs to be combined.
We solved this by using the Nearby Places search from the Google Places API, instead of a text search. This meant that we could directly feed in the user's coordinates to the API and get a place with real data out of it. This, however, lead to problem B: namely, at what radius should I set my tolerance to find the place associated with a geopoint. If we set it too small, we had issues of getting the business as the ATM or the pharmacy inside the store instead of the store itself, because the store's geopoint was too far away. If we set it too large however, other nearby buildings accidentally got sucked into the same batch of coordinates, so multiple stores in a strip mall may have been associated with the anchor store of the mall. Through trial and error, we think we found a radius that works for a lot of cases, but this is where the more detailed AutoML image analysis to determine what building the user is actually in comes into play, so we can control the radius dynamically per building in the future.
Accomplishments that I'm proud of
We are very proud that we were able to build this app since it was the first time we used Flutter to build a mobile app. In fact, we were able to implement all of the minimum viable product features that we wanted to make over the course of the hackathon, and integrates successfully between Google Maps, the Google Places API, and Google’s autocompleting search, Firebase, and Flutter. We really feel that the UI allows users feel comfortable within the app, and allow them to fluidly and naturally access the data.
What I learned
This was the first time using Flutter for both of us, so everything with respect to the Flutter Design itself was brand new. In fact, this was our first time doing any mobile app development, so while some of the Material design concepts were similar from working with React, things like navigation and page routes were entirely new.
This was also our first time using any of the Google Maps APIs in a project. Therefore, both interpreting what we needed and how to securely connect to that was also brand new. One important thing that we learned was that unlike most credentials, the API keys are acceptable to expose, you just need to restrict the access so they only work from within.
What's next for Crowd Scout
In the future, we plan to add the ability for businesses to sign onto the app directly and include their own numbers for maximum capacity.
We also plan to increase the accuracy of Crowd Scout by adding the ability to include people who are not using the Crowd Scout app in the population of a particular area by using Bluetooth signals.
Also, as mentioned in the how we built it section, if we have more time and resources, we'd like to retrain our AutoML model on a larger (and cleaner) data set so that we can use that to analyze if the user is in or out of a building to minimize the server updates that need to occur to update crowd numbers, and increase the accuracy of our count.
Finally, constantly updating the user's location is as the user is moving is also an issue that we started working on. Currently, it is built so the function call to identify where the user is occurs each frame the user moves. This was not a problem for our tabletop tests, as the emulator only moved location when we told it to, and even then the travel duration was negligible. However, we started thinking about how this would apply in the real world. Moving around inside of your own house or inside of the same store should not result in an update location database call. Likewise, moving too quickly in a single frame (i.e. you're in a car and thus we don't care what places you are near as you are not in any of them) should not result in updating the places database either. Therefore, to better improve computational efficiency, we would likely want to add parameters on the client side inside of the location update listener to only make the database places update call within a "donut" surrounding the user's last recorded point, to reduce processing of the aforementioned cases.
Our domain is CrowdScout.online.
Try It Yourself!
Everything is fully featured. Feel free to download the repository and try it out. You can run it through VS Code on either Windows or Mac (hypothetically, we didn't have a Mac to test ourselves). Just make sure you have Flutter installed on the machine, and the repo should do the rest.