Hello Folks, We know that in today’s world where there is a huge data that needs to be monitored. Suppose we have a website where a lot of people surf and that surfing create logs/metrics but think there will be huge logs/metrics. So is it possible to go through all this generated data in a thorough manner? Unfortunately, the answer is “NO”. But what if we can visualize this data !! That will be so exciting. So let's create this integration for visualizing the logs/metrics.
Firstly, let's understand some deep concepts about Kubernetes, Prometheus, and Grafana.
What is Kubernetes?
Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.
What is Prometheus and Grafana?
The combination of Prometheus and Grafana is becoming a more and more common monitoring stack used by DevOps teams for storing and visualizing time series data or logs. Prometheus acts as the storage backend and Grafana as the interface for analysis and visualization of that data.
What is EC2 Instance?
a virtual server service offered by AWS Cloud that offers thousands of images and hardware combinations for every imaginable use case.
What is Terraform?
Terraform is an infrastructure as code (IaC) tool that allows you to build, change, and version infrastructure safely and efficiently. This includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc. Terraform can manage both existing service providers and custom in-house solutions.
What is Ansible
Ansible is simply an open-source IT engine that automates application deployment, intra-service orchestration, cloud provisioning, and many other IT tools.
What actually we gonna do
- Launch Kubernetes Cluster using Kubeadm by Terraform and Ansible on the top of AWS EC2 Instances.
- Integration Of Prometheus & Grafana on top of Kubernetes.
- Prometheus Configuration and Grafana Database Storage should be persistent.
- Prometheus & Grafana Deployments should be exposed to the world.
- AWS account
- Terraform Setup
- Ansible Setup
How we built it
1. Create 3 EC2 instances on AWS Cloud using Terraform . one K8S master and two worker nodes
2. Configure K8S-Cluster using Ansible. K8S-Master
After that log in to the master node that is also configured as a client of k8s-cluster(
Type the command
sudo kubectl get nodes to view the worker nodes that are connected to the master.
3. Launch Prometheus Deployment This deployment is responsible for the Prometheus app which pulls the metrics from K8S-Cluster and also pulls from external exporters (node, app, service). Then store metrics in the Prometheus database which is persistent as it is mounted to external storage.
4. Launch Grafana Deployment This deployment is responsible for the Grafana app which pulls the data from the Prometheus database and shows the real-time status in graphical form in Dashboard.
_ Access Prometheus and Grafana through a browser using the Exposed ports and Master node Public IP. _
Challenges we ran into
This big challenge that I faced is I was not able to get the connectivity to the Kube-state-metrics is a listening service that generates metrics about the state of Kubernetes objects through leveraging the Kubernetes API; it focuses on object health instead of component health. This exposes useful information about node statuses, available deployment replicas, pod lifecycle statuses, certificate signing request statuses, and more. Another challenge was how can I store the Prometheus Database Data permanently, for solving that challenge I create a PV ( from the external source host, NFS, etc) then mounted the Prometheus Database on the PV which maintains data permanently even if the deployment crashed.
Shortened and Automated Workflows: Continuous deployment and development chain gets automated which makes the entire flow fast and improvised. Outbound Monitoring: A continuous check on dashboards for real-time projects with releases being made and defects getting fixed very easily. Customer Satisfaction: When an end customer gets a product with minimum bugs and defects, end customers’ faith and belief in your product will increase. Less time to Market: Automation makes the product release cycle less cumbersome as it involves fast deployments and continuous improvements.
Both Prometheus and Grafana are built around time-series data – with Prometheus primarily on the gathering side and Grafana on the reporting side. Both tools are open-source, are widely available with lots of community support, and are more than capable of meeting the needs of enterprises, large and small.