Managing Civo Kubernetes with terraform, kustomize and helm

Inspiration

I wanted to demonstrate an approach to managing Kubernetes clusters using terraform, kustomize and helm. Each cluster application is a terraform module, based on outputs from helm template, tweaked for purpose with kustomize, and tied together into two repositories

What it does

There are two key components:

• Build a Civo kubernetes cluster
• Populate the kubernetes cluster with real-world applications, with the ability to fully tweak those applications to best suit your needs

How we built it

The civo kubernetes cluster isn't really any more sophisticated than the standard civo terraform provider examples! We do manage a domain name as well as the cluster

For each application in the cluster, we populate the terraform module by running helm template, and then use the kustomization terraform provider to inject secrets, patch domain names and add any missing resources.

Challenges we ran into

• the civo webhook for cert manager couldn't add hosts with a dot (e.g. for adding _acme-challenge.application.example.com to the example.com DNS resource (worked around based on a new PR )
• the civo external-dns plugin hasn't yet been merged (worked around based on an existing PR )
• cilium doesn’t really work, I think because it’s incompatible with the flannel CNI - but I didn’t really need it
• ingress-nginx doesn’t play nicely with Traefik, which it doesn’t seem possible not to install (reported by @lefterisALEX as https://github.com/civo/terraform-provider-civo/issues/106) - again, I didn’t really need it as Traefik works well enough

Although the video stops before the applications were working in my demo, they worked a few minutes later with no further intervention on my part - just DNS being slow to propagate I expect!

Accomplishments that we're proud of

• HTTPS ingresses work - external-dns manages DNS correctly, and cert-manager is able to add the right records to allow LetsEncrypt to issue certificates
• prometheus is up and running with very few alerts!
• Five PRs against existing civo-related projects were added:
  • https://github.com/civo/terraform-provider-civo/pull/107
  • https://github.com/civo/terraform-provider-civo/pull/108
  • https://github.com/civo/terraform-provider-civo/pull/109
  • https://github.com/okteto/cert-manager-webhook-civo/pull/4/
  • https://github.com/okteto/cert-manager-webhook-civo/pull/5
• Log shipping is working - you can go to the docker-debug link and then view https://grafana.civo-hackathon.link/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Loki%22,%7B%22expr%22:%22%7B%7D%22,%22refId%22:%22A%22,%22range%22:true%7D%5D

What we learned

• Tying together grafana data sources is a real challenge

What's next for From zero to hero with Civo and terraform

• I'd like to better understand how to use node pools and firewalls to isolate internal from external traffic - being able to create a second, firewalled load balancer would be great
• Improve the monitoring links - while grafana was created, the integrations with tracing and logging aren't quite up and running
• Improve the documentation on how each module is created

Built With

• civo
• helm
• kustomize
• terraform