Much the same as many professionals are going through these days, my work project got scrapped in the mid of March, 2020, due to the impact of COVID-19 faced by the small and medium businesses. For almost a week I didn't have much to do on and while waiting for things to get back to normal, I began working on this project. The idea is to create a visualization of COVID-19 over these few months all around the world (and in the US) by gathering data on cases being reported and getting a sense of where we started to where we are going from then on.
In the hope that we'll all recover and recover well and soon from this.
What it does
It's a project showcasing data visualization on the latest COVID-19 cases reported around the world (and further drilling down to the US) to provide meaningful insights on how we are doing on dealing with the virus.
There is no actual metric but, with the timeline, we compare the rise in total cases from the previous day. We also show new cases/deaths by countries on the world map. The map is color coded using percentile distribution of case counts.
It shows a timeline of this data in a tabular form sorted by countries and allows the user to pin the table next to the map. We also show a timeline slider which begins from early Jan and goes till this day. A similar timeline data is viewable when drilling down to the USA. The data refreshes every hourly for showing the latest counts.
How we built it
The client uses React-map-gl which is a wrapper on top of map-gl to display numerical data on the world map/US map. An express based server is set to crawl/scrape data from online sources, parsed and cleaned before being fed into Redis using NodeJS backend.
Redis is used for storing the keys for each country and date which gets refreshed every hourly through crons which make the scraping and merging services to update Redis data. Nothing stops us from making the hourly updates to mins but it's not something that we planned on doing ATM.
Color coding and percentage calculations are calculated in the client. To make our app faster, we use Webpack and also pre-fetch some of the images. Webpack reduces our client build greatly and helps us serve the static assets faster on slower networks. We enable gzip compression for our service to further improve data transfer through the network. Our app is deployed on Heroku cloud service.
Challenges we ran into
Data collection was our first challenge. Having a reliable source and collecting data daily/hourly, in cost efficient way with having the flexibility to mold the schema was our first priority.
Data storage for a timeline view, deciding on the most effective schema consumable by the client and helping the it run faster was our second challenge as the geo data tends to become huge in size. This can introduce lags in the client making it jittery with calculating color coding on the fly that needed a schema customized to suit the needs better.
Filling the gaps of back dated data, recovering from the mishaps that happened during development were also some significant challenges. With respect to running the app, we run it using free dynos on Heroku and experience server cold starts but it's what you get with $0, so can't complain. :)
Accomplishments that we are proud of
Coming up with the concept, quickly putting the front-end and backend pieces and meeting the deadlines we set for ourselves is something that we celebrated after releasing our website successfully.
What we learned
We learned that dedicating time and effort to a project helps big time making thoughts into reality. We also learned how important it is to work together and achieve more as a team.
What's next for covid-visual
- Theming of the app to light and dark.
- Optimize on DB storage by compression at rest.
- Localization of the client.
- Mobile app