Using Ansible for AWS automation

Ansible setup and examples for managing cloud services.
By Kostas Koutsogiannopoulos

This tutorial assumes that you have ansible installed on a machine.
If not, check for installing it conviniently in a python virtual environment and running some basic administration tasks.

Identity information

In order to use ansible with amazon APIs we need to get API credentials provided by an Amazon EC2 Identity and Access Management (IAM).

So:

Connect to AWS console and click "Identity & Access Management" (category "Security and Identity").
Click "Users"
Choose your IAM user
Give your user the apropriate permissions from the tab "Permissions" (for example "AdministratorAccess")
Choose the tab "Security Credentials" and then choose "Create Access Key"

This will return somethind like that:

Access Key ID: XXXXXXXXXXXXXXXXXXXX Secret Access Key: XXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

These are the credentials you need for connecting to the AWS API.

You can set them as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables and continue on but we used boto configuration file as you will see.

Preparation of python environment

If you are using ansible inside a virtualenv, activate it and run:

pip install boto
pip install six

After installation create a file named boto.config and set the "BOTO_CONFIG" environment variable to point to that file.

Edit boto.config as follows (replacing eith your credentials):

boto.config

[Credentials]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

For our host inventory web can use the ec2 dynamic inventory script.

Download the files:

ec2.py
and
ec2.ini

Make the file ec2.py executable.

Set the EC2_INI_PATH variable to point to ec2.ini file.

Set the ANSIBLE_HOSTS variable to point to ec2.py file.

Example Playbook 1

In order to use the dynamic inventory (and the grouping possibilities that provides) we connect to AWS's console and tag our instance with this:

Name=example

Create a playbook like this:

ping_amazon.yml

- hosts: tag_Name_example
  tasks:
   - name: ping a server
     ping:

Run the playbook:

ansible-playbook -i ec2.py -u username --private-key=your_private_key.pem ping_amazon.yml

PLAY ***************************************************************************

TASK [setup] ******************************************************************* ok: [XXX_XXX_XXX_XXX]

TASK [ping a server] *********************************************************** ok: [XXX_XXX_XXX_XXX]

PLAY RECAP ********************************************************************* XXX_XXX_XXX_XXX : ok=2 changed=0 unreachable=0 failed=0

So in this example we used a dynamic inventory, we question the inventory with a specific tag (Name=example) and ansible-ping the server returned by the inventory.

Example Playbook 2

In this example we will add a new inbound rule to an existing security group. Security group is basicaly the amazon's firewall.

By default there is no firewall rule registered and the server is unaccessible from the outside world.

Say that you have a web server that replies to all requests on ports 80 and 443. You already created a security group with some rules to achive that and now you want to connect to that machine via SSH and do some configuration.

This is the playbook:

open_aws_ssh.yml

---
- name: Add a security group to the instance
  hosts: tag_Name_example
  connection: local
  tasks:
    - name: SSH
      local_action:
        module: ec2_group
        name: ssh
        region: xx-xxxx-x # The amazon region hosting your server
        description: This is a security group generated by ansible
        rules:
          - proto: tcp
            from_port: 80
            to_port: 80
            cidr_ip: 0.0.0.0/0
          - proto: tcp
            from_port: 443
            to_port: 443
            cidr_ip: 0.0.0.0/0
          - proto: tcp
            from_port: 22
            to_port: 22
            cidr_ip: XX.XXX.XXX.XXX/32  # Replace with the ip of your console machine

Run it with:

ansible-playbook -i ec2.py open_aws_ssh.yml

As you can see we created the rules from scratch. Ports 80 and 443 for http, https access from the world and port 22 from our IP only.

After a successful run we can access the server via ssh.

Example Playbook 3

We created the next playbook in order to give you an idea about how we can configure services running on our servers via ansible.

This playbook is not related exclusively to aws cloud.

We create a local directory containing configuration templates for all the services we need to manage.

In order to show you that, we got a sample configuration from (nginx web server), we replaced domain1, domain2 with {{ domain_1 }}, {{ domain_2 }} creating a template file named template_nginx.conf and we saved locally on our ansible control server.

template_nginx.conf

user www www; ## Default: nobody
worker_processes 5; ## Default: 1
error_log logs/error.log;
pid logs/nginx.pid;
worker_rlimit_nofile 8192;

events {
worker_connections 4096; ## Default: 1024
}

http {
include    conf/mime.types;
include    /etc/nginx/proxy.conf;
include    /etc/nginx/fastcgi.conf;
index    index.html index.htm index.php;

default_type application/octet-stream;
log_format   main '$remote_addr - $remote_user [$time_local] $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
access_log   logs/access.log main;
sendfile     on;
tcp_nopush   on;
server_names_hash_bucket_size 128; # this seems to be required for some vhosts

server { # php/fastcgi
    listen       80;
    server_name {{ domain_1 }} www.{{ domain_1 }};
    access_log   logs/{{ domain_1 }}.access.log main;
    root         html;

    location ~ \.php$ {
      fastcgi_pass   127.0.0.1:1025;
    }
}

server { # simple reverse-proxy
    listen       80;
    server_name {{ domain_2 }} www.{{ domain_2 }};
    access_log   logs/{{ domain_2 }}.access.log main;

    # serve static files
    location ~ ^/(images|javascript|js|css|flash|media|static)/ {
      root    /var/www/virtual/big.server.com/htdocs;
      expires 30d;
    }

    # pass requests for dynamic content to rails/turbogears/zope, et al
    location / {
      proxy_pass      http://127.0.0.1:8080;
    }
}

upstream big_server_com {
    server 127.0.0.3:8000 weight=5;
    server 127.0.0.3:8001 weight=5;
    server 192.168.0.1:8000;
    server 192.168.0.1:8001;
}
server { # simple load balancing
    listen          80;
    server_name     big.server.com;
    access_log      logs/big.server.access.log main;

    location / {
      proxy_pass      http://big_server_com;
    }
}
}

Now we can use the "template" ansible module to create a configuration file on our remote servers, based on domain_1 and domain_2 variables.

After nginx configuration file creation you probably need to reload nginx service.

Here is the playbook:

create_nginx_conf.yml

---
- name: Create nginx configuration file
  hosts: tag_Name_example
  become: true
  vars:
    domain_1: example1.com
    domain_2: example2.com
  tasks:
  - template:
      src: "/home/pi/ansible_env/ansible/config_templates/template_nginx.conf"
      dest: "/home/username/project/example_nginx.conf"
      owner: username
      group: username
      mode: 0644
  - service:
      name=nginx
      state=reloaded

...and we are running it with:

ansible-playbook -i ec2.py -u username --private-key=your_private_key.pem create_nginx_conf.yml

Example Playbook 4 - The last playbook

At last we can automate the procedure of making an image from an instance running on the cloud. The new image can be used to create another instance - replice of the first one, for development reasons for example.

Here is the playbook:

instance_replicate.yml

---
- name: Replicating the production server
  hosts: tag_Name_example
  connection: local
  tasks:
   - name: Replicating the production server
     local_action:
       module: ec2_ami
       instance_id: i-xxxxxxxx # The instance id you want to replicate
       region: xx-xxxx-x # # The amazon region for AMI creation
       wait: yes
       name: test_image

The cloud may be need some time to create the new image, so you can change the playbook with:

... wait: no ...

...if you do not want your ansible job to waid for the image to reach the "available" status.

Clarifications

You are not encouraged to "over-use" amazon proprietary services for your environment creation, configuration, replication etc. because you will find yourself commited to a specific cloud provider before you realize it.

Instead, try to use portable services that will be functional on any cloud provider.

For example try to work with virtual environments (like linux containers) that you can transfer from server to server without modifications to the default operating system.

A future tutorial will give you a complete example about configuration management with this mentality.