Civic Data Service Mesh with DRP & CORTX

This project leverages an open source service mesh along with CORTX storage to offer a framework for public data exchange. Static and streaming data sets can be advertised and consumed.

Comment

Inspiration

In February 2021, our area was hit by ice and snowstorms not seen in 60 years. As a member of a local off-road club, I helped transport medical workers and bottled water to local hospitals. The most challenging part of that week wasn't the road conditions - it was the lack of fresh data. We struggled to coordinate transportation and deliveries over social media.

But... what if we'd had a platform which allowed resource providers and consumers to better coordinate? What if there were a public, civic data mesh that allowed for a faster exchange of information?

What it does

DRP (Declarative Resource Protocol) is an open source service mesh project originally designed for use with infrastructure sources. Microservices declare static and streaming resource definitions which are searchable across the mesh. It is a perfect pairing for CORTX storage due to the object oriented nature of the record sets.

How we built it

After learning about S3 storage (first timer!) I used the aws-sdk npm module to interact with a local CORTX VM instance. The DRP service modules had to be updated to use S3 storage from Mongo. A handful of microservices representing civic data sets were added to the demo script and rolled into a Docker container.

Run Docker Container

Run the adhdtech/drp-nodejs container with "node serverCortx.js". Substitute the S3ENDPOINT, S3ACCESSKEYID and S3SECRETACCESSKEY values with your 

> docker run --name drpcortx -P 8080 -e REGISTRYURL=ws://localhost:8080 -e PORT=8080 -e S3ENDPOINT=https://192.168.5.148 -e S3ACCESSKEYID=AKIAT7y9JwKbREyMsCX08G3blg -e S3SECRETACCESSKEY=IHQRZ8U/06tboLFyLduRdLfWvkSt9gl1rHQUeATQ adhdtech/drp-nodejs node serverCortx.js

The service should now be listning on port 8080 for you to access with your browser.

Challenges we ran into

The biggest challenge was the lack of search functions in the CORTX S3 API. Migrating from Mongo resulted in search functions being implemented in the microservices.

Accomplishments that we're proud of

A functioning demo showing that is it feasible for civic data providers to publish data sets to a public mesh.

What we learned

CORTX storage will make it much easier to develop microservices which can transition between local and cloud storage.

What's next for Civic Data Service Mesh with DRP & CORTX

Recently I've been discussing the creation of a public data mesh with local civic data experts. In the future we hope to see a POC that incorporates static and streaming datasets from local municipalities.

Built With

Share this project:

Updates