Web Server

CloudFormation template: yaml

AWSTemplateFormatVersion: 2010-09-09
Description: A web server running on an EC2 instance

TODO

Forward application logs to CloudWatch (journald or file?)
Note where to look for logs on instance
Can cloud-init, cfn-init, or cfn-hup logs fill up? Do they need to be rolled?

Overview

This CloudFormation template creates an EC2 instance running a web server. The web server is a golang binary stored in S3.

Automatically start the service with systemd on the instance.

Install the CloudWatch Logs agent, and configure it to send the web server logs to CloudWatch.

CICD Process

Here is a CI/CD process that creates a deployment for every git branch.

Push commit to repository
CICD commit builds artifact and pushes to S3
S3 object located at $BUCKET/$commit
- Should there maybe be folders per $branch?
CICD runs a cloudformation update
- Updates $BuildArtifactKey in AWS::CloudFormation::Init
  
  TODO: export but do not tangle
```
    config:
      files:
        /home/ec2-user/go-webserver:
          source: $ARTIFACT_URL
```
On the ec2 instance, cfn-hup eventually runs and realizes there was a change to AWS::CloudFormation::Init (specifically, the above artifact URL changed)
- cfn-hup runs and pulls down the new artifact

Prerequisets

../network/public.html deployed with the same DeploymentName
An EC2 KeyPair in this region

Parameters

Parameters:

Param	Value
DeploymentName	webserver
KeyName	cfc
SSHCIDR	0.0.0.0/0
BuildArtifactBucket	my-s3-bucket-1n9japign3xm7
BuildArtifactKey	main/go-webserver-amd64

DeploymentName

  DeploymentName:
    Type: String
    Description: A name for this deployment

A deployment is a deployed application, potentially comprised of many CloudFormation stacks. This is sometimes called an "environment", but that is an overloaded and confusing term. Use the DeploymentName to indicate which logical deployment a stack belongs to.

If a deployment is completely specified by exactly one CloudFormation template, the DeploymentName and the AWS::StackName refer to the same things. In that case, consider not using a DeploymentName parameter.

KeyName

  KeyName:
    Type: AWS::EC2::KeyPair::KeyName
    Description: The name of an EC2 KeyPair
    Default: test

AWS does not provide a CloudFormation type for EC2 KeyPairs. This is a deliberate choice: a KeyPair consists of a public and private key, and AWS does not want to expose the private key in CloudFormation outputs.

This is one example where Terraform has an advantage: since Terraform runs locally, it can create a KeyPair and save the private key locally.

In CloudFormation, you have two options:

Create the KeyPair beforehand, then provide the KeyName to your CloudFormation template as a parameter.
Create the KeyPair using a CloudFormation Custom Resource (i.e. a Lambda Function) defined in your CloudFormation template. The Custom Resource must save the private key somewhere (like SecretsManager or SSM Parameter Store).

SSHCIDR

  SSHCIDR:
    Type: String
    Description: IP CIDR range

Allow SSH access from this CIDR range.

BuildArtifactBucket

  BuildArtifactBucket:
    Type: String
    Description: Name of an S3 bucket with the web server build artifacts
    Default: test-s3-bucket-bucket-1n9japign3xm7

The S3 Bucket where the web server artifact is stored.

BuildArtifactKey

  BuildArtifactKey:
    Type: String
    Description: S3 object key for a web server build artifact

An S3 object key which identifies a particular object in the BuildArtifactBucket. Values for this key will likely be a git commit hash, or perhaps a git tag name, so that an artifact can be associated to a particular git commit.

Mappings

Mappings:

Instance mapping

  RegionMap:
    us-east-1:
      AMI: ami-0aeeebd8d2ab47354
    us-east-2:
      AMI: ami-0d8d212151031f51c

Resources

Resources:

IAM

The EC2 instance needs access to:

CloudFormation, so cfn-init can describe this CloudFormation stack
CloudWatch Logs, so the cloudwatch-agent can write logs to CloudWatch
S3, so cfn-init can retrieve the web server binary

Role

  InstanceRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${DeploymentName}-InstanceRole"
      Description: Allows EC2 instances to call AWS services
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: CfnInit
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - cloudformation:DescribeStackResource
                  - cloudformation:SignalResource
                Resource: "*"
        - PolicyName: GetBuildArtifacts
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                Resource: !Sub "arn:aws:s3:::${BuildArtifactBucket}/*"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Instance Profile

  InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      InstanceProfileName: !Sub "${DeploymentName}-InstanceProfile"
      Roles:
        - !Ref InstanceRole

Security Group

  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow HTTP and SSH
      GroupName: !Sub "${DeploymentName}"
      VpcId: {Fn::ImportValue: !Sub "${DeploymentName}-VpcId"}
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: !Ref SSHCIDR
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: !Ref SSHCIDR
      Tags:
        - Key: Name
          Value: !Sub "${DeploymentName}-instance"

Instance

In this case we are using a combination of UserData and the AWS::CloudFormation::Init section (which the cfn-init program reads) to define how to provision the EC2 instance.

AWS says this is a best practice, and this works ok for small examples, but defining all provisioning logic in AWS::CloudFormation::Init definitely has some disadvantages:

Length. Makes the CloudFormation template really long.
Limits. You may run into CloudFormation limits:
- max template length (51,200 bytes)
- max length of the AWS::CloudFormation::Init section
Only AWS. The provisioning language is specific to CloudFormation, so you can't use the same YAML to provision, say, a vagrant machine or an Azure VM (as opposed to using something like Ansible which could provision VMs anywhere).
Sluggish updates. cfn-hup does not run immediately after a cloudformation update. Instead, it runs periodically (configurable with the interval option). So several minutes may go by after a cloudformation update before cfn-hup re-runs cfn-init, which in turn re-provisions the instance. Considering how complicated this setup is, you'd think it would at least be snappy and event-based, not poll-based.

Even in this simple case, the CloudFormation gets pretty long with information that might otherwise be specified using a configuration management tool (like Ansible, Salt, or Chef).

Some alternatives to using only AWS::CloudFormation::Init:

Use cfn-init to download a git repo or gist with provisioning scripts. Then re-run the provisioning scripts after every CI/CD build by updating a flag file in the AWS::CloudFormation::Init section (e.g. a file who's name is the git commit hash).
Specify the provisioning logic in AWS Systems Manager Run Command. Then use an EventBridge event to kick off the Run Command
- I haven't done this
- Can those scripts be managed in git?
- Also kind of nice you can manually kick of an AWS Run Command thing.
Use a CM tool (e.g. Ansible, Salt).
- It could either run from your CI/CD runtime or even on the instance itself, if the instance downloads the Ansible configs.

  Instance:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Authentication:
        S3AccessCreds:
          type: S3
          buckets:
            - !Sub "${BuildArtifactBucket}"
          roleName: !Ref InstanceRole
      AWS::CloudFormation::Init:
        configSets:
          all:
            - 01_install_cfn_hup
            - 02_install_cloudwatch_agent
            - 03_install_web_server

          # Configure and start the cfn-hup daemon, which checks CloudFormation
          # AWS::CloudFormation::Init section every $interval and then performs
          # the actions in /etc/cfn/hooks.d/ hook files.
        01_install_cfn_hup:
          files:
            # A config file for cfn-hup (can it use IMDS for region?)
            /etc/cfn/cfn-hup.conf:
              content: !Sub |
                [main]
                stack=${AWS::StackName}
                region=${AWS::Region}
                verbose=true
                interval=2
              mode: "000400"
              owner: root
              group: root

            # Tell cfn-hup to rerun cfn-init after every cloudformation update
            /etc/cfn/hooks.d/cfn-auto-reloader.conf:
              content: !Sub |
                [cfn-auto-reloader-hook]
                triggers=post.update
                path=Resources.Instance.Metadata.AWS::CloudFormation::Init
                action=/opt/aws/bin/cfn-init -v \
                    --region ${AWS::Region} \
                    --stack ${AWS::StackName} \
                    --resource Instance \
                    --configsets all
                runas=root
              mode: "000400"
              owner: root
              group: root

            # cfn-hup systemd unit file
            /lib/systemd/system/cfn-hup.service:
              content: |
                [Unit]
                Description=cfn-hup daemon
                [Service]
                Type=simple
                ExecStart=/opt/aws/bin/cfn-hup
                Restart=always
                [Install]
                WantedBy=multi-user.target

          commands:
            01_enable_cfn_hup:
              command: systemctl enable cfn-hup
            02_start_cfn_hup:
              command: systemctl restart cfn-hup

        02_install_cloudwatch_agent:
          files:
            /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json:
              content: |
                {
                  "metrics": {
                    "append_dimensions": {
                      "ImageId": "${!aws:ImageId}",
                      "InstanceId": "${!aws:InstanceId}",
                      "InstanceType": "${!aws:InstanceType}"
                    },
                    "metrics_collected": {
                      "mem": {
                        "measurement": [
                          "mem_used_percent"
                        ]
                      },
                      "swap": {
                        "measurement": [
                          "swap_used_percent"
                        ]
                      }
                    }
                  }
                }

          commands:
            01_stop_cwa:
              command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop
            02_start_cwa:
              command: |
                /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
                    -a fetch-config \
                    -m ec2 \
                    -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json \
                    -s

        03_install_web_server:
          files:
            # Download the web server binary from S3
            /home/ec2-user/go-webserver:
              source: !Join
                - ""
                - - https://
                  - !Ref BuildArtifactBucket
                  - !Sub .s3.${AWS::Region}.amazonaws.com/
                  - !Ref BuildArtifactKey
              mode: "000755"
              owner: root
              group: root
              authentication: S3AccessCreds

            # Create a systemd unit file for our web application
            /lib/systemd/system/go-webserver.service:
              content: |
                [Unit]
                Description=The go-webserver application
                [Service]
                Type=simple
                ExecStart=/home/ec2-user/go-webserver
                Restart=always
                [Install]
                WantedBy=multi-user.target

          commands:
            01_enable_webserver:
              command: systemctl enable go-webserver
            02_start_webserver:
              command: systemctl restart go-webserver

    Properties:
      IamInstanceProfile: !Sub "${DeploymentName}-InstanceProfile"
      ImageId: !FindInMap [RegionMap, !Ref "AWS::Region", AMI]
      InstanceType: t2.nano
      KeyName: !Ref KeyName
      SecurityGroupIds:
        - !Ref SecurityGroup
      # SsmAssociations:
      #   - SsmAssociation
      SubnetId: {Fn::ImportValue: !Sub "${DeploymentName}-PublicSubnet1"}
      Tags:
        - Key: Name
          Value: !Sub "${DeploymentName}-instance"
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          yum update -y
          yum install -y \
              aws-cfn-bootstrap amazon-cloudwatch-agent
          # Run cfn-init which processes the AWS::CloudFormation::Init section
          /opt/aws/bin/cfn-init -v \
              --region ${AWS::Region} \
              --stack ${AWS::StackName} \
              --resource Instance \
              --configsets all

Outputs

Outputs:
  PublicIP:
    Description: Public IP of the EC2 instance
    Value: !GetAtt Instance.PublicIp
    Export:
      Name: !Sub "${DeploymentName}-PublicIP"