Python virtualenv management with Ansible

Using Ansible to replicate the virtual environment of a Django web application.
By Kostas Koutsogiannopoulos

In a previous post we described how we can use Ansible to manage cloud services, for example, to replicate a server instance living in AWS and use it for development - testing purposes.

Of cource this was an example that can present you some options but heavy using of cloud provider proprietary services can lead you to expensive and limited solutions.

Ideally we want to use only portable environments that can move from server to server and from a cloud provider to another easily with minimal infrastructure or operating system configuration.

Following this mentality along with the persistence to do every configuration with a configuration management tool like Ansible can help our application to be more portable, scalable, easy to replicate for testing, development etc.

This post is about a real working python virtualenv containing a Django application with all its dependencies. The consept is the creation of an isolated test environment running on the same linux server. In addition to the application we need to replicate the database because our testing plan may be include schema changes or other data migration.

So we need to automate this scenario:

Connect to the production server.
Compress the application in a package.
Backup the application package in our local server.
Extract the package to the remote server in the test workspace.
Create a new python virtualenv.
Install the python packages that our application depends on, in our new virtual env using a "requirements" file.
Configure the application for test by changing debug status and database name in settings.py.
Create a new "staging" database in the same instance as a production replica (you may need to become the instance owner to do this).
Check that everything is ok via "manage.py check" command.

The scenario is reflected to the following ansible playbook:

create_dev_environment.yml

- hosts: tag_Name_example
tasks:
   - name: Compress production project
     command: tar -zcf production_project.tar.gz /prod/project
   - name: Getting the compressed file locally
     fetch: src=/prod/production_project.tar.gz dest=/home/user/backup flat=yes
   - name: Create the remote test project
     unarchive: src=/home/user/backup/production_project.tar.gz dest=/test/dev_project
   - name: Create the remote virtualenv
     pip: virtualenv_python=python3.4 virtualenv=/test/test_env requirements=/dev/dev_project/prod/project/requirements.txt
   - name: Change django setting for DEBUG mode and database connection
     replace: dest=/dev/dev_project/prod/project/application/settings.py regexp='^DEBUG = False$' replace='DEBUG = True' backup=yes
   - replace: dest=/dev/dev_project/prod/project/application/settings.py regexp=''NAME'[:] 'prod_db',$' replace=''NAME'{{':'}} 'stage_db','
   - name: Replicate the database
     become: 'true'
     become_user: 'database_instance_owner'
     command: createdb -O owner_user -T prod_db stage_db
   - name: Check the application
     django_manage: command=check app_path=/dev/dev_project/prod/project virtualenv=/test/test_env

Of cource, our source code may be managed by a git repository. So instead of copying files, we can hit our repository server every time we want to get a version of our code.

This is an example Ansible playbook that implements version HEAD checkout from a remote repository -via SSH- to a local repository:

checkout_project.yml

- hosts: tag_Name_repos
tasks:
   - name: Ping the server
     ping:
   - name: Checkout HEAD
     local_action:
       module: git
       repo: ssh://repo_srv_user@reposrv/home/repo_srv_user/project
       dest: /home/localuser/development
       accept_hostkey: yes
       key_file: /home/pi/reposrv_private_key.pem

A working example like that can give you an idea about the processes we can automate and run easily again and again or run against multiple servers with no effort using Ansible.