Next Generation Networks with Telemetry Data

Inspiration

Infrastructure as a Service (IaaS) Platform provided a level playing field allowing small and medium-sized companies who don't have upfront investment capabilities. Networking has been hit with a sway of industry changes. We hear about a lot of buzz words about Software-Defined Networking, OpenSource, DevOps Practices. These open a new path not just to the top companies who are constantly competing to stay top at the market, but to those who now can use opensource technologies to give these big companies a run for their money. As a Team of Network Engineering Students, we wanted to explore some of these buzz words. Put into to use a robot cloud platform like Google Cloud to provide “Telemetry and Distributed Scaling” as a Service.

What it does & How we built it

Any IaaS Service provider provides a rich set of metrics to evaluate the performance of servers that they host. Since they have a robust methodology and well-defined templates for this, can these IaaS Providers market this? In this project, we tried to build a Telemetry Pipeline hosted in the IaaS environment. The clients push telemetry data to collectors(we are using GCloud MySQL Servers) distributed in the IaaS platform. (Of course, the IaaS already takes care of the inter-networking of these collectors.) We are running Google Cloud Functions that periodically check for new data in these collectors. Based on the service signed up by the client ( here due to time constraint we only did 2 servers, client infrastructure CPU measurement and auto-scaling based on the CPU). We took these data and feed them into an InfluxDb Time Series Database. We further Visualized the data with Grafana. Once we have some data, we tried to gather intelligence from the collected data. If the CPU Utilization was more than a predefined threshold , we used the Grafana interface to alert the network admin. We didn't stop there. We triggered a Jenkins build to check if the client has the allotted Quota to spin up new instances. If the Client has Quota we went on to provision these new servers for them. If they don't have sufficient Quota we will revert back to the network admin.

Challenges we ran into

Updating the GCP Firewall policy to have required access for our IaaS service platform running on GCP.
Integration of SQL, InfluxDB, and Grafana.
Limited experience with the containerization with Docker
Running and configuring the Jenkins inside a Docker

Accomplishments that we're proud of

We have an end to end product working.
We were able to learn containerization with Docker in deep and were able to resolve all the challenges we faced.
We were able to have a complete CI/CD pipeline implemented on Jenkins.
Integrating all the components together.