Using Ansible on EC2 with sensitive data

The problem

I use Ansible to configure my instances on EC2. I often needto include passwords, SSL keys, and other sensitive data that doesn't belong in source control. AWS allows you to include user-data when launching an instance. You can include arguments to your instance in the user-data field (as JSON for example), or you can include a script to bootstrap the instance - but not both. Since I use Ansible to configure an instance from scratch, I use a user-data script to install Ansible.

The problem is actually a little more complicated because I launch my EC2 instances from withina CloudFormation template, and I use Fabric to generate and run the CloudFormation template.

Additionally, I have to abuse Ansible to use it in my workflow. Ansible is easy to dive in to, butone of the first things you do is define an Inventory, which is basically a list of hosts to be managed with Ansible.

For example:

[webservers]
foo.example.com
bar.example.com

Great, right? No. This is a problem for me because I never know the IPs of machines I plan on configuring beforehand, because they are all provisioned on demand ( I don't want to manage my instances centrally anyway). Ansible includes an EC2 Inventory script which claims to solve this, but really just introduces more problems. One big problem, is that Ansible queries information about instances many times when running a playbook, so many times in fact that the EC2 inventory script has to include a cache that stores instance information locally in a file. The documentation actually recommends a cron job to refresh the cache. Now, complexity has been added and we no longer get current information about instances. We can do better than this.

I almost gave up on Ansible at this point, but then I discovered Pull mode playbook, which is a shortcut for cloning a git repository of playbooks and executing one whose name matches <server-name>.yml or local.yml. This is a step in the right direction, but there are still a couple of problems with this approach. I either have to make a <instance-ip>.yml file for every instanceI create in my playbook repository, or I create a separate repository for every type of instance with a local.yml in it - in which case I don't get to reuse any of my Ansible code.

It turns out that you can run a playbook locally:

$ansible-playbook -c local /mnt/ansible-playbooks/my-role.yml

Now we are getting somewhere. What I need now is a way to stage and retrieve my playbooks, and pass in to the instance which playbook it should use.

Workflow Overview

1. Commit changes to an application to be used in a development stack.
2. Stage the application code.
3. Create a CloudFormation template with options (whether or not to use RDS, for example).
4. Stage private data
5. Use the CloudFormation template to create a running stack.
6. Instances created within the stack install Ansible, download playbooks and self configure.

How it all works

Starting from the beginning, I use git for my source control. I need to stage my application code so that the application can retrieve it later. You might be tempted to use GitHub, but don't. GitHub is a great service, but it does go down from time to time. Instead, use S3. I use private GitHub repositories for my source code, but I mirror them in S3. Here is how to do it with Fabric.

# fabfile.py
# requires Fabric and awscli
from fabric.api import local

def upload_s3():
    """Stage to S3"""
    local('tar czf /tmp/my_project.tar.gz .')
    local('aws s3 cp /tmp/my_project.tar.gz s3://my-bucket/git/my_project.tar.gz')
    local('rm /tmp/my_project.tar.gz')

I also keep my Ansible playbooks in a git repository, so I can use the same code above to stage my playbooks.

For CloudFormation, I use Fabric and Troposphere. I use Troposphere to generate the templates for CloudFormation because I don't like writing large JSON documents by hand, and my templates change frequently. Here is a simplified example:

# fabfile.py
from troposphere import FindInMap, Join, GetAtt, Base64, Parameter, Ref, Template, ec2
from fabric.api import local

def make_template():
    """Generate a CloudFormation template"""
    # handle options, etc.
    # ...
    # also not shown are including AWSRegionArch2AMI, AWSInstanceType2Arch maps

    # The top level CloudFormation Template
    template = Template()

    # Adds an EC2 Instance to the template
    instance = ec2.Instance(
        "MyInstance",
        ImageId=FindInMap(
            "AWSRegionArch2AMI",
            Ref("AWS::Region"),
            FindInMap(
                "AWSInstanceType2Arch",
                Ref(instance_type),
                "Arch"
            )
        ),
        IamInstanceProfile='my-role', # Used for EC2 roles
        InstanceType=Ref(instance_type),
        SecurityGroupIds=Ref(server_security_group_ids),
        SubnetId=Ref(server_subnet),
        KeyName=Ref(keyname),
        Tags=[
            ec2.Tag("Name", Join("", [Ref("AWS::StackName"), "-instance"])),
        ],
        UserData=Base64(Join("", ansible_bootstrap_src))
    )
    template.add_resource(instance)

    # Write the template to a file
    with open(output_file, "w") as outf:
        outf.write(template.to_json())

    # Create a stack from the template
    local("aws cloudformation create-stack --stack-name {0} --template-body file:///{1}".format(stackname, output_file))

The variable ansible_bootstrap_src is the a script provided as user-data to the instance, and is used to install Ansible and execute a playbook. I have written it in Python, so that I can inject variables into it.

HEADER = """#!/bin/bash -ex
# Bootstrap ansible onto an EC2 instance
# All output is logged to stdout and ~/log.txt

# Install Ansible and dependencies
APT="apt-get install -y -q"
$APT python-paramiko python-yaml python-jinja2 python-simplejson git-core 2>&1 | tee ~/log.txt
"""

FOOTER = """
# ANSIBLE_URL, PLAYBOOK_URL, and ROLE must be defined

wget $ANSIBLE_URL -O ansible.tar.gz
tar xfz ansible.tar.gz
cd ./ansible
source ./hacking/env-setup

echo "localhost" > ~/ansible_hosts
export ANSIBLE_HOSTS=~/ansible_hosts
wget $PLAYBOOK_URL -O /mnt/ansible-playbooks.tar.gz 2>&1 | tee ~/log.txt

mkdir /mnt/ansible-playbooks
tar xfz /mnt/ansible-playbooks.tar.gz --directory /mnt/ansible-playbooks 2>&1 | tee ~/log.txt
ansible-playbook -c local /mnt/ansible-playbooks/${ROLE}.yml 2>&1 | tee ~/log.txt

exit 0
"""

Now I can provided URLs to the script from Python. But, I can also provide other environment variables, such as my Django SECRET_KEY.

# If these values contain a shell character, and you don't quote
# them, you're gonna have a bad time.
exports = [
    "export PLAYBOOK_URL='", playbook_url, "'\n",
    "export ANSIBLE_URL='", ansible_url, "'\n",
    "export SECRET_KEY='", django_secret_key, "'\n",
]

Finally, we can combine our bootstrap script with our variables:

ansible_bootstrap_src = HEADER.splitlines(True) + exports + FOOTER.splitlines(True)

At the end of my Ansible playbook, my application is started in the same environment as the playbook - so that the environment variables it needs are present.

Sensitive Files

The approach above works fine for passwords, string keys and such - but what about files? To begin with, I store them in private S3 buckets. One easy way to give bucket accessto an EC2 Instance is through IAM Roles.I don't want to provide persistent access to my private bucket however, I just want to give the instance access long enough to provide access to what it needs on startup. Luckily, S3 has a mechanism for pre-signing URLs which included a timeout. This allows you to generate a URL that can be passed to the instance on startup, but will expire.

# requires python-boto
from boto import connect_s3

conn = connect_s3()
cert_url = conn.generate_url(600, 'GET', bucket='my-private-bucket', key='my-role/ssl/my-ssl-cert')
exports = [
    "export PLAYBOOK_URL='", playbook_url, "'\n",
    "export ANSIBLE_URL='", ansible_url, "'\n",
    "export SECRET_KEY='", django_secret_key, "'\n",
    "export SSL_CERT_URL='", cert_url, "'\n",
]
ansible_bootstrap_src = HEADER.splitlines(True) + exports + FOOTER.splitlines(True)

Summary

Using this workflow, you can provision stacks with CloudFormation (without writing any JSON by hand), have all the instances in them self configure, and provide them with sensitive data. Additionally, you aren't relying on any services outside of AWS.