-
-
11k points in this view, but it could also be millions.
Inspiration
As a user, too many times one sees Google Maps doing wonderful stuff, but extremely limited in terms of the quantity of data displayed. There is a simple reason for this. Mapping APIs are in-memory based on each browser (including smartphones). We, at EveryListing.com, are a worldwide company presenting real-estate properties. As such, people expect an interactive mapping system, yet they also do not expect constraints against the magnitude of data displayed.
What it does
Our system implements a distributed mapping logic where the heavy-lifting happens on the back-end. That way, the RAM limitation is lifted and the user-experience opens-up to interactions with an almost limitless amount of points.
How we built it
In the javascript implementation, points and clusters are dynamically set following an https communication to a servlet in java. Such servlet in java keeps many highly optimized indexes per zoom level. This in fact pre-calculates clusters at the different zoom levels, ensuring the response-time is always fast. The index structures in use are hybrid in RAM and on storage, dynamically loading and unloading based on need and caching for next usage up to a given limit in RAM. This allows for an almost immediate resolution of any request without the need to load the entire context in RAM and this limits the indexing system based on storage space and not RAM in the backend.
Challenges we ran into
Responsiveness: Every user action needs to be responded way within the sub-second range. Scaling correctly the servlet and putting our indexing structures on SSD within the Google Cloud infrastructure allowed us to typically serve any action within 200ms.
Concurrency: Modifying the data-sets (point) required the recalculation of indexes in such a way that cache information had to be ignored for future calls. As such, there is a little lag between a change in data and its way to the user's display in cases where the data was cached. That being said, it is not a significant problem for our current domain of concern (real-estate), but it could be if anyone else would want to implement a 911 map for example (time-sensitive information expected in that case).
Accomplishments that we're proud of
Bringing any user-experience to a fully usable outcome is always challenging, yet we've achieved it. Any error condition also needs to be handled gracefully from the customer's perspective, and this was also achieved. It is also within the realm of this implementation to display millions or even 10's of millions of points within a single map, and that was never achieved before and the source of lots of achievement.
What we learned
One needs to invent as one wants to move forward. Although APIs are offered, one cannot limit itself to the natural use of such APIs and expect an exceptional outcome. To achieve some level of greatness (not saying we've made it there ourselves, you are the judges), one needs to rely on APIs but not exclusively. Creativity around the APIs is necessary.
What's next for Big-data geographical interaction system
Many aspects are currently being improved. For one, the actual project displays a uniform set of points that are not categorized. An improvement of this is in progress where the points are categorized. Furthermore, we are also consuming 7 billion categorized geographical points to search and display them dynamically. Since our distributed system is not limited by RAM, we expect these 7 billion points to reside on the backend and allow for real-estate searches such as "find properties at most 5km from a Marina, at most 15km from a golf course, with a high-density of restaurants and pubs" and it should come back with worldwide matches in the sub-second range.
Log in or sign up for Devpost to join the conversation.