doctl app spec validate-offline

Contributed a new command to DigitalOcean's CLI tool

Comment

Inspiration

While working on docker-phasik.tv, I had the idea to try and validate a DigitalOcean App Schema YAML file as a CI/CD or local linter step. I had been using pre-commit for such things in the past, so I decided to try and implement this as a pre-commit hook. Along the way, I found that doctl always wanted an API token, and made API requests for the normal app spec validate command.

I did not want to expose any DigitalOcean API tokens in the GitHub Actions CI/CD environment. However, the pre-commit hook I had developed only worked when passed a token and after running doctl auth init. This led me down the rabbit hole towards making changes to 3 other projects to implement this new offline mode App Spec validation feature.

What it does

doctl app spec validate-offline is a new doctl sub-command used to validate DigitalOcean App Spec files (e.g. .do/app.yaml).

The upstream doctl currently provided a app spec validate command. However, it had the following limitations:

  • doctl app spec validate:
    • Required an API token, and access to the internet to contact DigitalOcean's API
    • Internal Go code runs the equivalent of doctl app spec propose as part of the checks run when --schema-only is not passed.
    • When --schema-only is passed, the command still required an API token. With this flag, the YAML is only checked against Unmarshalling the data structure schema. Yet, the way that GoLang Cobra commands were implemented had the limitation of always setting up an API client.
  • doctl app spec validate-offline: I submitted this brand new command to doctl in digitalocean/doctl#1450
    • Does NOT require an API token 🎉
    • Does NOT require internet access to validate an App Schema either! 🎉

The new offline mode was used in the following dependent pull-requests:

How we built it

I first started down the rabbit hole of what it might take to implement simple App Spec validation. This led me to first build a proof-of-concept hook using the usual doctl app spec validate. I then realized that this would require passing API tokens and potentially exposing those secrets in the CI/CD environment. After that realization, I ended up making changes to the action-doctl project in order to make passing the token and running doctl auth optional.

After those parts were implemented, I realized that the underlying doctl command always required an API token to have been setup, even for the presumably offline --schema-only validate command. Once that issue was found, I worked on implementing a secondary validate-offline command inside doctl.

Putting all the pieces together produced a wonderful result! A new CI/CD workflow that could validate the DigitalOcean app.yaml file schema without an internet connection or API token required.

Challenges we ran into

This prototype hook required changes in 3 or 4 different languages (if counting GitHub actions YAML). doctl is written in Go, action-doctl is written in TypeScript/JavaScript, and the pre-commit-digitalocean hooks are written in POSIX shell.

Accomplishments that we're proud of

Everything works together nicely! This should also close a few issues across these projects once all the pull requests are merged.

What we learned

Underlying the first problem to solve are sometimes 3 or 4 other problems (and projects) that need changes for a fully working solution to be put together. Lots of moving parts means lots of testing each piece.

What's next for doctl app spec validate-offline

  • Code reviews from DigitalOcean
  • Merging PRs
  • Switching docker-phasik.tv back off of pre-release PoC branches to use those merged PRs once they make it to the main branches

Built With

Share this project:

Updates