Avalanche

Inspiration

As a Production Engineer its always a challenge to automate application stack bootsrapping from scratch which involves provisioning of the server, dns entry updates, Cluster/loadbalancer udpates, app setup per Env etc. We're trying to solve this with a complete automation in place using opensource technologies.

How it works

When we provision a server, we configure it to run a post install with some necessary params which is required in further steps of making this server fit for production. In brief, these are the steps

1. Post install triggers after server boot and register its details in Zookeeper.
2. Monitor Daemon scans the zookeeper data tree and upon finding new node entries, Initiates an app installation setup. We provide salt formulas for each of the app stack deployment. A salt client runs on master less mode on the server to bring up the app stack.
3. Once app installation successfull, monitor deamon moves new node entry active node tree, with attributes for deployment status and monitored(nagios) status.
4. Monitoring Daemon ensures addition of the node to nagios.

Challenges I ran into

1. Random failures/errors we ran into during the hack.
2. Handling monitoring automation part.

Accomplishments that I'm proud of

We could achieve a good progress on complete automation with just one day. We realised that this can be made even more beautiful with improved error handling

What I learned

Learned more about zookeeper, salt, vagrant,

What's next for E2E server onboarding automation

*DNS entry addition automation *Choose a better inventory management software (with apis) to store the states of servers.