During my master's degree this year, I had an opportunity to do cloud infrastructure automation as a part of my Master's Thesis at a leading automotive company. Working with real-time DevOps systems was very fascinating for me and I continue to explore various DevOps tools and cloud ops technologies. Then I started making simple automated tools to make my daily life easy. I made my own CI/CD for my pet projects following various tutorials. Day by Day, it becomes super exciting and finally, I decided to pursue a career in DevOps and automation.
What it does
Whenever a new commit is made, the system automatically packages and pushes the application to the docker hub enabling continuous integration. Automated chef InSpec test case execution on Kubernetes configuration files, is also triggered by the GitHub action as a part of continuous integration. Once the docker image is available in the docker hub, the application is automatically deployed into the Kubernetes cluster. Helm charts are used for making Kubernetes configuration easy. The project is integrated with Argo CD to enable Continuous Delivery for both Staging and production environments. Other ways to package the application are also explored via docker habitat. The artifacts of a sample app have been created using chef habitat and exported as a docker image to demo the use case. More detailed documentation is available in Github. Also, Slack has been integrated into the project, thus whenever a new commit is made, slack notification is pushed.
How we built it
Initially, a clear project plan and roadmap has been made via Trello Board and Spreadsheet. More details about the plan sheet are in the read me section of the Github repo. In this project BrewOps, A sample web app called Automateforgood is used to exhibit the best DevOps practices. Whenever a new commit is made to the GitHub repo, Github Actions automatically triggers an action to package the sample application(app name is Automateforgood) as Docker Image and push it to the Docker Hub enabling continuous integration. Another Github Actions trigger the system to execute Chef InSpec test cases over Kubernetes configuration files as a part of continuous integration. The Kubernetes cluster is provisioned using K3s in a vagrant box where the application can be deployed. Once the docker image is available in the docker hub, it is automatically deployed into Kubernetes Cluster. Kubernetes Manifest template is made using Helm Charts and input configuration files for Staging and prod environment are made. ArgoCD is used to enable Continuous Delivery on each deployment at the Staging/Prod Environment. In the end, I have also explored the advantages of chef habitat and experimented with it as it is quite handy when it comes to cross-platform builds. Created a Docker image from artifacts created by docker habitat for a study purpose.
Challenges we ran into
Choosing a tech stack was bit difficult because I have to compare and contrast the advantages of one over the other. For Example, Github Actions vs Jenkins, Kubernetes vs Docker Swarm, etc. I had a lot of environmental issues as I work on multiple projects so took this as an opportunity to utilize vagrant to enable a single workflow for the project. Also, I didn't find the inbuilt GitHub actions for the chef Inspec so it was a bit tricky to set up my own chef Inspec action script. The configuration errors caused due to the variety of YAML files are very minute which took a lot of time to debug the kubernetes & helm configurations as a beginner. I am very new to chef habitat so felt it was a bit challenging to set up things. Finally, I explored ELK vs Prometheus, Grafana, datadog integration, etc but not completed the observability part due to time constrain and other schedules.
Accomplishments that we're proud of
- End to End exhibition of CI / CD system.
- Having solid project plan (Trello/Excel attached to github) to finish the project on time.
- Very clear commit messages throughout this long hackathon cycle.
- Built custom GitHub Action to execute chef inSpec and dockerisation.
- Explored multiple DevOp tools/techniques in a short period of time.
What we learned
I have learned the list of best DevOps practices to be followed in the industry. Though I have never used chef stack before, after this hackathon project, I got good hands-on with Chef Inspec and Chef Habitat. Found that Chef Habitat is quite handy when it comes to cross-platform builds. Found Chef Inspec as a very friendly tool to use. Learned to use vagrant environment. I have written my own GitHub actions scripts and got comfortable writing them anytime in the future as well. My YAML configuration debugging skills has improved a lot. I really enjoyed learning/writing Kubernetes configuration files and helm charts. I have integrated GitHub with slack and found that we can do this fancy step with just a few configuration changes. I have explored ArgoCD which is quite easy to set up. I took a lot of time to choose one stack over another so got expose to a lot of key tools like ELK, Prometheus, Grafana, datadog integration, etc.
What's next for BrewOps
- Add more Chef InSpec Testcases to completely test/ audit the applications and infrastructure.
- Utlize chef cookbooks for installing and setting up kubernetes cluster.
- Configure indepth log levels to the application to enable easy debugging.
- Explore application of Prometheus for monitoring and Grafana, for Observability.
- Explore Polaris to ensure that Kubernetes pods and controllers are configured properly utilizing best practices.
- Explore more about the real-time use cases of Chef Habitat. Already I experimented with the usage of chef habitat by building the artifacts of a sample application from scaffoldings and exported it as a docker image. You can see the execution here. am looking forward to knowing more about the real-time use cases Chef Habitat.
- Build CI/CD Pipelines in AWS/Azure and explore the world of Cloud Ops. I am consistently self-learning and you can find my CI/CD pipeline deployment over the Azure project here. Looking forward to keeping the momentum.
- Explore the application of chef automate in application, security, and infra automation.