The CloudFormation Layer Cake

So I've had the idea of this post for a while. Like "3 years ago" a while:

I first came across the idea of the CloudFormation Layer Cake from this AWS Advent blog post from 2012, so it's not a new idea, and definitely not mine.

While I've been sitting on this blog post draft, it's been interesting to see how well the idea has stood the test of time. Additions to the service like CloudFormation Imports/Exports, support for SSM Parameters as template parameters, etc. have all enhanced the pattern. I think the reason why it is still relevant speaks to it capturing some basic, best-practices when writing and designing software (including infrastructure-as-code) in general.

The Layer Cake 🍰

But first, what is the CloudFormation Layer Cake? It - like cake - is simple and delicious.

Instead of putting all your resources in one CloudFormation stack, split them up in to logical groups that reflect their interdependencies, and layer them on top of each other to form your environment/application.

Here's a visual example:

And the same resources deployed according to their rate of change, dependencies, and logical grouping:

Benefits

While the layer cake approach satisfies my craving for food-based technology metaphors, it also has a lot more tangible benefits which make it a good idea. You'll notice that none of these are really unique to CloudFormation, they're just good ideas in general.

Explicit Dependencies

Creating layers of resources, and selectively using outputs from lower layers in the higher layers forces you to declare you dependencies explicitly. This forces you to think about interdependencies sooner rather than later, when it's hard/impossible to change them.

Ordering

While CloudFormation generally does a good job of managing the ordering of creating your resources, there are a few edge cases where you have to use DependsOn to have resources be provisioned in the right order. With a layered approach ordering is explicitly enforced since you are deploying resource in separate stacks in a specific order.

Loose Coupling

When you are forced to define your inputs and outputs clearly, you reduce the coupling between parts of your system and make yourself more aware of them at the same time. This also makes it harder for unintentional/hidden dependencies to sneak in. The result is a systems that are easier to troubleshoot, as the "touch-points" between components (where things often go wrong) are clearly defined and understood.

Limit Blast Radius

Using layers means that when you make a change you already know that you can only directly impact other resources in that layer, rather than all your resources at the same time; I adhere to the adage configuration changes caused more outages than code ever did!

Running the deploy command against a stack with no changes will tell you as much, and do nothing to the running stack (as opposed to nested stacks, see below).

Deployment Speed

A beneficial side-effect of reducing the blast-radius of change means you're deploying less resources at once, which just so happens to mean that your deploys will be faster! If you're worked with AWS for a while, you'll appreciate anything which stops you from having to deploy a multi-AZ RDS instance, update a CloudFront Distribution (this is not too bad these days), create certificates, etc.

Security

Related to the blast radius aspects are the security improvements possible with this approach: You can limit and control access to specific resources by putting them in a protected layer. For example in an environment with a lot of developers you could enforce a strict isolation of IAM resources in to their own layer, creating them in advance and only allowing developers to use them, not create/update them.

Implementation

When it comes to implementing the CloudFormation Layer Cake there's a few different options to do so natively in AWS; they each have their pros and cons, so should be considered based on your scenario:

Import/Export Stack Outputs

This approach has a number of traits:

• Dependencies are declared/checked at stack deployment time
• Resources/stacks that are depended on cannot be deleted

Systems Manager Parameter Store Parameters

Yes, I used the word "parameter" twice there. Using Parameter Store Parameters as CloudFormation template parameters is a thing, just remember:

• Parameters are evaluated at deployment time; if a parameter changes (in Parameter Store) later, you will need to do an UpdateStack to receive the new value
• You can view evaluated values of the parameters in the stack info

Just don't forget that currently you can't use SecureStrings as template inputs 😭

Nested Stacks

The layer cake approach is in direct contrast to the CloudFormation nested stack approach, which ironically the service also supports natively. I used to consider the nested stack approach an anti-pattern in CloudFormation, and something to be removed if found; this was mainly due to the potential of a child stack getting in to an unrecoverable state (like UPDATE_ROLLBACK_FAILED), putting the parent stack in to failed state that left you with no option but to redeploy 😱

Relatively recent changes to CloudFormation (like the ability to continue a rollback) have made nested stacks a less-bad approach (but still not desirable). The fact that you can protect your resources, and also adopt resources in to a stack means that you can now theoretically recover from any scenario, but it's not something I would recommend or rely on... Even if it works it be very painful.

I still don't like the fact that an update to a stack with nested stacks requires an update of all the stacks.

Layers are Good

So go forth and build!

But seriously, I encourage you to try this next time you get the chance. Like all techniques it takes practice to get good at, but the benefits are worth the effort.

Wait, How Many Layers?

On some levels (see what I did there 😏) this approach also imitates how we build physical structures; start at the bottom, and progressively add things on top; You wouldn't build a house starting with the second floor, you'd put down a solid foundation first.

Deciding on how many layers, and where to draw the line is where the art comes in to it; How much do you split your stacks up in layers? The answer (for better or worse) is: As much as makes sense, but not more than that!
Usually I aim to let my dependencies drive out my layers.

As you practice you'll get more comfortable with making the decision to create a new layer or add to an existing one. You'll probably never get it perfect, and with the benefit of hindsight you'll nearly always do something differently, and that's OK.

One way to practice it to review an existing system/application, and think how you'd layer it if you had the chance to do it again.

Just be conscious of your approach, apply best practices where the opportunity presents itself, and leave things better than you found them.

Photo by Free To Use Sounds on Unsplash