I am very interested in IoT. I truly believe it is the future of computing, especially with edge computing on the rise. As a result, we are going to have tons of data for processing and post-analysis. Fluentd can act as a log collection point and output it to the highly scalable, CORTX.
What it does
This is an integration of which Fluentd runs as a daemon on a machine. It looks for data sources given the input you choose and outputs it to CORTX via S3-type matching in Fluentd.
How I built it
I essentially used the VM provided by Seagate on Cloudshare and hooked up the endpoints after creating the configuration file for Fluentd. I tested the writes and reads to CORTX via the AWS CLI.
Challenges I ran into
It is very easy to misconfigure something. I spent nearly 3 days trying to write a custom plugin for Fluentd to support CORTX. After I came across MinIO (another S3 compatible storage) being used with Fluentd, I knew for sure I had messed up my configuration and that is indeed what had happened. The other issue I had is that the Fluentd documentation is very young and needs more care. I noticed things that seemed off and different in some places. Not sure if those are errors, but I will definitely be submitting PRs for those by creating examples in the documentation for others to use.
Accomplishments that I'm proud of
I successfully integrated a popular CNCF tool with CORTX and made my first pull request in a large project like CORTX.
What I learned
Alot of things can go wrong. Just keep trying it out :)
What's next for CORTX-Fluentd Integration
I'm going to launch Fluentd as part of the EFK stack and output my results to CORTX.