Infrastructure as Code in AWS

A guide to get started with Infrastructure as code tools in AWS.

Raywon Kari - Published on July 13, 2020 - 12 min read

---

Introduction

Infrastructure as Code (IaC), is a practice which enables the use of automation to manage resources declaratively such as provisioning, configuring, updating, de-commissioning etc. More theory around this topic can be found on wikipedia. IaC practises have gained a lot of popularity because it brings lot of consistency, predictability, reusability into the system.

Since this is the cloud era at the moment, cloud providers more or less have given the same ability to use IaC practises to manage cloud infrastructure, and IaC in AWS is powered by a service called CloudFormation. Using it, we declaratively define cloud resources in either JSON 📃 or YAML 📃 templates and deploy them to AWS. Rest is taken care by the service itself.

In this blog post, we will explore basics of cloudformation, how to use it, some tips and tricks and a few examples.

---

Basics of CloudFormation

When we go out to a restaurant, what do we usually do 🍛 ?

We tell what we want to eat, we get the food 😄 and at the end we pay the bill.

That is exactly what is happening with CloudFormation as well. We tell what we want in the templates, we upload the template, and it creates and configures the stuff for us. So writing the template is ordering the food, creating and configuring the resources is getting the food, and paying the bill here is the common aspect 😉.

Following diagram is an example of how CloudFormation (CFN) works.

Summary:

• CFN expects a template in JSON or YAML
• We can upload the template directly from a computer or from an S3 bucket
• CFN then creates a stack, which consists of the resources based on how it is declaratively defined in the template
• Using CFN is free of cost, we only pay for the payable resources it creates.

---

Playing with CFN Templates

Template is the heart of CFN service. Therefore it is very important to understand the format and how to write a template. A template can consist of the following sections:

• Format Version => the CFN template version that this template comforms to. This is identifies by the capabilities of the template.

• Description => description of the template.

• Metadata => objects that provide additional information about this template.

• Parameters => values which the template has access to, during runtime. Can also be termed as variables.

• Mappings => mapping of constant values such as a key-value pairs, which are used in conditional scenarios.

• Conditions => conditions that controls the behavior of resources and their execution.

• Transform => transform is primarily used in serverless scenarios, where we could use SAM (serverless application model) syntax for instance.

• Resources (required) => specifies all the actual AWS resources to be created as part of this stack and its necessary configuration.

• Outputs => values which are to be exposed, thereaby allowing external tools to fetch this exposed data.

Out of all the sections, Resources section is the only mandatory section which needs to be present and at least one resource should be defined in any particular template.

Template structure:

Here is an example structure of a template in JSON format:

{
  "AWSTemplateFormatVersion" : "version date",

  "Description" : "JSON string",

  "Metadata" : {
    template metadata
  },

  "Parameters" : {
    set of parameters
  },

  "Mappings" : {
    set of mappings
  },

  "Conditions" : {
    set of conditions
  },

  "Transform" : {
    set of transforms
  },

  "Resources" : {
    set of resources
  },

  "Outputs" : {
    set of outputs
  }
}

Here is an example of an actual usable template:

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "AWS CloudFormation Sample template showing how to create a publicly accessible S3 bucket configured for website access with a deletion policy of retain on delete.",       

  "Resources" : {
    "S3Bucket" : {
      "Type" : "AWS::S3::Bucket",
      "Properties" : {
        "AccessControl" : "PublicRead",
        "WebsiteConfiguration" : {
          "IndexDocument" : "index.html",
          "ErrorDocument" : "error.html"
         }
      },
      "DeletionPolicy" : "Retain"
    }
  },

  "Outputs" : {
    "WebsiteURL" : {
      "Value" : { "Fn::GetAtt" : [ "S3Bucket", "WebsiteURL" ] },
      "Description" : "URL for website hosted on S3"
    }
  }
}

In this template, we are simply creating an S3 bucket configured to act as a website with read access from internet and also a deletion policy, meaning if the CFN stack is deleted, it will not delete the s3 bucket. And in the outputs section, we are exposing the website URL.

I am using AWS CLI throughout. For instructions on how to install and configure the CLI refer here.

Lets create a file named stack.json and copy the template to it. In order to deploy the template, lets use the following command:

aws cloudformation create-stack --stack-name mytestwebsite \
--template-body file://stack.json --profile raywon

# Once the command succeeds, we should see an output like this
{
    "StackId": "arn:aws:cloudformation:eu-west-1:XXXXXXX:stack/mytestwebsite/xxxxxxxxxxxxxxxx"       
}

In the template section description above, I mentioned that all the outputs are exposed outside CFN, so any service can fetch the data from outputs. Lets try to query the data using describe-stacks CLI call as shown below:

aws cloudformation describe-stacks \
--stack-name mytestwebsite --profile raywon

This command will give a JSON response as shown below:

{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:eu-west-1:xxxxxxxxxx:stack/mytestwebsite/xxxxxxxxxxxxxxxxxxx", 
            "DriftInformation": {
                "StackDriftStatus": "NOT_CHECKED"
            }, 
            "Description": "AWS CloudFormation Sample template showing how to create a publicly accessible S3 bucket configured for website access with a deletion policy of retain on delete.",       
            "Tags": [], 
            "Outputs": [
                {
                    "Description": "URL for website hosted on S3", 
                    "OutputKey": "WebsiteURL", 
                    "OutputValue": "http://mytestwebsite-s3bucket-v5t8i1hj340y.s3-website-eu-west-1.amazonaws.com"
                }
            ], 
            "EnableTerminationProtection": false, 
            "CreationTime": "2020-07-11T10:03:21.690Z", 
            "StackName": "mytestwebsite", 
            "NotificationARNs": [], 
            "StackStatus": "CREATE_COMPLETE", 
            "DisableRollback": false, 
            "RollbackConfiguration": {}
        }
    ]
}

So here we can see the outputs objects with the data which we wanted to be exposed of. Now lets try to create an index.html & error.html files which are configured to be served as part of the S3 website.

I use very minimal stuff as shown:

#This is index.html
<h1>
    This is index
</h1>

#This is error.html
<h1>
    This is error
</h1>

Once the files are uploaded, make them public and our website is ready. Head over to the http endpoint which you see from the outputs section to access the website.

Detailed documentation on templates can be found here.

---

Updating CFN Templates

So far we have seen how to use templates and create resources. In this section, lets explore how to update resources by updating templates and executing them.

In order for us to see what we are updating, before executing the changes, we will use a feature called change sets. CFN allows us to see what the changes are, before actually executing them.

So in our earlier example, lets simply update the bucket name. From the stack.json I will add the bucket name property. Here is how the resource section looks like now:

"Resources" : {
    "S3Bucket" : {
      "Type" : "AWS::S3::Bucket",
      "Properties" : {
        "AccessControl" : "PublicRead",
        "WebsiteConfiguration" : {
          "IndexDocument" : "index.html",
          "ErrorDocument" : "error.html"
         },
         "BucketName": "my-test-bucket-name-raywon"
      },
      "DeletionPolicy" : "Retain"
    }
  },

Now that our new template is ready, lets try to generate a changeset. I use the following command:

aws cloudformation create-change-set --stack-name mytestwebsite \
--change-set-name update-bucket-name \
--template-body file://stack.json --profile raywon

# we get a JSON response as follows
{
    "StackId": "arn:aws:cloudformation:eu-west-1:xxxxxxxx:stack/mytestwebsite/xxxxxxxxxx",       
    "Id": "arn:aws:cloudformation:eu-west-1:xxxxxxx:changeSet/update-bucket-name/xxxxxxxx"
}

Now to see the changes, we need to look at changeset again, as follows:

aws cloudformation describe-change-set \
--stack-name mytestwebsite \
--change-set-name update-bucket-name \
--profile raywon

# I get a JSON response with changes.
"ResourceChange": {
    "ResourceType": "AWS::S3::Bucket", 
    "PhysicalResourceId": "mytestwebsite-s3bucket-v5t8i1hj340y", 
    "Details": [
        {
            "ChangeSource": "DirectModification", 
            "Evaluation": "Static", 
            "Target": {
                "Attribute": "Properties", 
                "Name": "BucketName", 
                "RequiresRecreation": "Always"
            }
        }
    ], 
    "Action": "Modify", 
    "Scope": [
        "Properties"
    ], 
    "LogicalResourceId": "S3Bucket", 
    "Replacement": "True"
},

Here we can see that, a new bucket will be recreated. So we can go ahead executing these changes. Once the changes are executed, we need to copy the index.html and error.html in to the new bucket, and make sure to delete the old bucket manually because we have the deletion policy enabled.

aws cloudformation execute-change-set \
--stack-name mytestwebsite \
--change-set-name update-bucket-name \
--profile raywon

After this change is done, I have a new website using the name of the newly created bucket accessible http://my-test-bucket-name-raywon.s3-website-eu-west-1.amazonaws.com.

So we have now seen an example on how to use CFN to spin aws resoruces and also to execute changes to those resources,

Summary:

• Once a template is deployed, always prefer creating change sets to verify the changes.
• Once we are satisfied with changes, execute the changeset.

Miscellaneous Info:

• For each CLI command we executed earlier, make sure to review the config and data from the AWS Console. This will help to get more feel around the console, as well as get more hands on with what we are doing here.
• So far we have used AWS CLI to deploy and update stacks, we can achieve the same operations by using the AWS Console as well.
• We can also use several SDKs to operate with Cloudformation. All AWS SDKs can be found here.
• Most of the times, we need to create custom templates, and writing pure JSON or YAML can become a cumbersome activity. Therefore many people either generate templates using open source tools such as troposphere, or take reference templates and modify them.

---

Further Reading

Summary:

So far we have seen the overview of working with CloudFormation, such as:

• Creating a CFN template.
• Deploying the template to CFN.
• Creating & Executing template changes using changesets.

Here are some other resources which can help gain more thorough understanding about CFN and its capabilities:

• CloudFormation Designer => Using this tool, we can visually prepare an architecture, and as a result, we will get a template as output. We can also view and edit existing templates using designer.
• API Reference => CFN API reference can be found here. In here, we can see the full list of operations we could do with CFN.
• Sample Templates => In here, we can explore various sample templates AWS have created for us.
• CFN Resources => Here is a list of all AWS services we can manage using CFN.

---

Tips & Tricks

This will be last section of this blog post, and in here we will explore some of the tips and tricks when using CFN.

Description :

• Please use the Description property very extensively everywhere possible.
• It helps for new comers or for someone who reads the template for the first time understand what this is about.

Naming :

• Do not name resources with flowery words or language.
• Stick to ARNs, or logical names which CFN creates.
• If we want to spin up multiple instances of the same resource, it will fail if we statically name them due to naming conflicts.
• Use tags if you MUST have custom names because of compliance or some other dependencies.

Using IAM to control access :

• CFN is a very powerful service which can create, at the same time destroy resources.
• When using CFN to provision resources, use dedicated IAM roles for stacks designated to perform only necessary actions against relevant resources.
• Embrace least privilege access model.
• Some info on this can be found here.

Keep track of limits :

• All AWS services have limits, some of which are soft, and some are hard.
• CFN also has limits. For example, we can launch 200 CFN stacks per region by default.
• Doc on this can be found here.

Nested Stacks :

• Whats more happier if we could break down a huge task into smaller doable tasks.
• Similar is the ideology with nested stacks. A huge project's resources are broken down into components and they form building blocks of that project.
• Each component is a stack of its own which are inter dependent on each other to form a sequence of steps. Use it if the project is growing.
• This involves thought process upfront as well. If you think you will create more than 10 resources in a stack, break it down into nested.

Re-use templates :

• If the same set of resources needs to replicated in different accounts or environments, use the same template as source.
• Setup automation so that, once the template is updated, the automation should deploy the same changes one by one across environments/accounts.

You can find all the code used in this blog post here .

If you have any questions/thoughts/feedback, feel free to contact me in any platform you prefer.