SLAINTE

Inspiration

Validation of server configuration is not something we do very well. Currently, we rely on Puppet to run it's own validation and fix mistakes. However, we do not verify the configuration beyond that and trust the Puppet has left things the way our code describes. While this serves us well, it does not provide a fast mechanism for verify the expected configuration in a fast or automated way. Finding ways to quickly verify the expected configuration of a host and the ability to extend that mechanism will provide new insight into the state of our infrastructure and reduce time to recognize issues.

What it does

SLAINTE transmogrifies the catalog of configuration found on a host running Puppet into configuration for the server testing and validation application Goss. It provides a Puppet resource for defining Goss tests for resources which are not found in the Puppet catalog. Finally, it provides systemd services to run and manage Goss as an HTTP endpoint /healthz for either internal or external health checks.

How I built it

Using jq to create a Goss configuration, a systemd service will generate the configuration file before starting the goss service. Puppet was used to manage all of the resources and provide the interface for creating tests not discoverable in the Puppet catalog.

Challenges I ran into

Building on a personal POC done in August, the maintainability of Goss templates with the variability of the Puppet catalog was very difficult and required traversing the catalog many time. To improve this, the transformation from Puppet catalog to Goss configuration was re-written in jq. The first pass has created a slightly longer, but much more flexible script which generates the same tests.

The next challenge was finding a reliable way to manage multiple instance of Goss running on a host as some of the checks and be slow, and may not be required to verify that a server is in the state expected. This was attempted using systemd template services and environment files, but time was an issue to get this part tested (with reviews due, a skip level in the middle of the day, and family obligations over the weekend).

Accomplishments that I'm proud of

Transforming the goss templates into a jq script with runs 5 times faster and is significantly more readable is a good accomplishment. Also building out the scaffolding for running multiple instances of the end point to allow for more finely scoped testing is a point of particular pride.

What I learned

Adapting a one off test is not as simple as expected. Challenge days where other company duties are required result in almost no flow time to work on the project.

What's next for SLAINTE

Finishing and polishing the ability to run multiple instance of goss on a host such that various sets of configuration can be validated quickly will allow for more complete health checking of hosts which can be largely completed in an automated or reliably coded way with direct visibility in to the relationship between server configuration, the application of the configuration and the resulting validation of the expected configuration.