CloudFormation templates: continuous testing for infrastructure as a code

Question

On the project we use some of AWS services like AWS Lambda, EC2, AWS API Gateway, ElastiCache, etc. Also we have CloudFormation template which describes whole our infrastructure. As the project is developed we begin to use some new AWS services or change configuration of some which are already used. Also with that we should to keep our CloudFormation template up to date.

And here we face with issue that we need to be sure that our CloudFormation template is valid, correct and that we can use it for creation of infrastructure if it will be needed. In such case we need something like continuous testing for our template. Which approaches are more appropriate for that?

Should we configure automatic creation of stack from our CloudFormation template as part of continuous integration process and to track template changes in our repository? Or there are better solutions?

Answer 1

You can do some simple validation of CloudFormation templates using the aws cloudformation validate-template CLI command. This is roughly the equivalent of static code analysis for other languages: it checks things like parameter name typos and that the template is syntactically valid JSON/YAML; but is quite limited in terms of what validation it can perform.

As that article says, the only sure-fire way to check that a CloudFormation template will create resources the way you want/expect it to is to try it, and that does indeed mean creating stacks as part of your CI and testing process. Since this can be slow in the case of some resources and expensive in the case of others, you may want to limit the commits on which the full stack-creation testing is performed.

Answer 2

We have been using cfn-python-lint as a precursor to building. Should this fail, we do not build. Rules provided in cfn-python-lint are a lot more comprehensive than aws cloudformation validate-template and in addition, it gives you some good practices rules and it also gives you a framework to write your own rules (which we use for governance).

Additionally, we don't build on feature branches, we build only master. We gives devs an environment to play with where they can run the pipelines that we would normally run on master and in dev/staging/prod. This is a completely separate account where they have just about full reign. This obviously isn't fool proof as our sandbox area may not reflect what's in dev/staging/prod since people play with it, but it's helped us a lot.