Useful Custom Datree Policies for K8s Admins

To tackle the issues I've come across at Paytm Insider quite often when deploying to Kubernetes, I've leveraged multiple property paths feature of Datree to max to create custom policies.

Comment

Inspiration

I'm a Software Engineer at Paytm Insider, handling Paytm's Movies vertical, where we use Kubernetes extensively. We have a separate repository where we store our helm charts and other manifest files. However, we do not do any static analysis on these files currently. When I learned about Datree when this hackathon was announced, I was quite excited to create custom policies for the issues which I have faced at Paytm Insider along with exploring the other best practices that we can follow.

What it does

I've created multiple custom policies with rules majorly targeting the following use cases:

  • cost reduction
  • governance
  • security
  • stability

These rules involve asserting the following best practices:

  1. Ensure each container has a configured CPU request within range
  2. Ensure each container has a configured CPU limit within range
  3. Ensure each container has a configured memory request within range
  4. Ensure each container has a configured memory limit within range
  5. Ensure Ingress only uses approved domain names for hostnames
  6. Prevent containers from running without a read-only root filesystem
  7. Ensure containers do not allow privilege escalation
  8. Ensure containers do not run processes with root privileges
  9. Ensure containers do not expose sensitive host system directories
  10. Prevent containers from running on the same node if multiple replicas are specified

How we built it

hub.datree.io acted as a starting guide for me. I really appreciated how easy it was to follow and get started with datree. As soon as I came across multiple property paths example, I knew I had what I needed. I started with the rules for resource quotas and continued from there, thinking about the past challenges I've faced. Swagger docs came in handy to understand the YAML format in a much better sense.

Challenges we ran into

Figuring about the regex pattern for detecting exposed host system directories which are sensitive was quite difficult. The original regex I came up with passed in Datree's YAML Validator, but didn't pass when I ran it with datree's cli. After investing a few hours, I finally figured out the regex, where Noaa Barki was datree team helped in verifying from her end. It was a good time trying to bypass every new iteration of the regex we came up with. :D

Accomplishments that we're proud of

I'm really proud that Cloud Native Hackathon is my first hackathon, as it has introduced me to several new tools and folks.

What we learned

  • Dah! Datree and its many many use-cases
  • Several new K8s practices especially from a security and cost's point of view
  • More familiarity with regex
  • More familiarity with YAML

What's next for Useful Custom Datree Policies for K8s Admins

I've already learned how to integrate these policies in Github Actions for our repository at Paytm Insider. Hence, the next step will be to use these to set up a pipeline and leverage this to prevent the issues I've across.

Built With

Share this project:

Updates