From the beginning of the pandemic, we were obsessed with providing consistent views of cases, death and test results in all countries, especially comparing our home country (Colombia) and the United States.
Making use of Open-Source tools like Zabbix and Grafana, as well as by leveraging AWS Serverless capabilities, we were able to achieve this with remarkable accuracy and consistency between different data sources.
What it does
Our COVID-19 Visualization application uses a dynamic content automatically fetched from AWS Data Exchange subscriptions (just like REARC repositories) to S3 buckets. From there, data points are normalized, validated, compared and presented to end-users via publicly available Grafana dashboards.
How we built it
The core application runs on a Lambda function built on Python which captures events generated in Cloudwatch whenever a new data-set revision is made available by the provider (in this case, REARC Testing data and New York Times COVID-19 data) and exports it to a S3 bucket.
Once that data is pushed into the S3 bucket, a new Lambda function parses those metrics and pushes each one into a Zabbix Server landing zone where the automatic pre-processing engine from Zabbix loads that data into its database for alerting and exposure.
Finally, all data in consumed by Grafana via an API gateway that provides great response time for drill-drown, refresh ratios and overall interaction with each widget available in the dashboard.
Challenges I ran into
Making sure that we deal with different refreshment rates from data sources, catching errors on data transcription (cities or states with no changes in 24-hours) and pushing more and more countries into our repository to improve the data normalization rules within Zabbix and AWS Lambda.
Accomplishments that I'm proud of
Enabling automatic ingestion from AWS Data Exhange subscriptions using Cloudwatch events, AWS Lambda and S3 buckets. Creating Grafana dashboard templates that can be extrapolated to other similar time-tracking healthcare data relatively easy.
What I learned
The great potential in AWS Data Exchange and the agility that it will bring to data-driven applications where external data availability was a major roadblock in improving these types of solutions.
What's next for our Dashboards
Adding more countries and heterogeneous data sources. Improve our data normalization rules.