Extending root fs of the whole farm

On a previous article we described in detail how we can resize the root partition of a single linux server using interactive tools like fdisk.

In this article we will use Ansible to automate the whole procedure and run it against a number of linux centos servers. For our demonstration we created a cloud-like environment based on Virtualbox including 12 servers created from the same centos image. We assume that all virtual disks are already resized from 6GB to 10GB (VBoxManage modifyhd <virtual_disk_file> --resize 10240) and all the servers are up 'n running. From this point on, we will complete the resize live (without reboot the virtual machines).

Ansible inventory

The inventory is a simple file, listing hosts in groups. In our example we have 12 servers in a group named "DOZEN-VM":

 inventory

On every server we have registered a "configuration management" user with root privileges and username "ansible".

We have also configured ssh login without password using ssh keys on all servers. For your reference:

$ ssh-keygen

$ ssh-copy-id ansible@centos7_XX

At last we have authorized the user "ansible" to gain privileges without password using visudo and the dollowing entry:

ansible ALL=(ALL) NOPASSWD: ALL

Now lets run a simple test to verify that we have conectivity with all of our servers.

$ ansible -i inventory -m ping -u ansible DOZEN-VM

The playbook

The playbook bellow is following the logic:

1. Create a new partition on device
2. Extend volume group on the new partition (creating physical volume on it)
3. Extend LVM
4. Extend filesystem

We assume that root filesystem lives as a whole on the last disk partition so we only need a second parttion starting at the end of the first and ending at the end of the disk.

In order to accomplish that using the existing ansible modules (parted) we need to get information of the current partitions and use it to create the new one. To be more specific, we need the exact end of the last partition on the disk. This will be the starting point of the new partition.

Take a look at the first task where we get the required information (partinfo variable).

At the second task we use this information accordingly in order to create the new partition:

partinfo.partitions[-1].endKiB: The ending in Kilobytes of the last partition

partinfo.partitions[-1].num: The number of the last partition (for example 2 for /dev/sda2)

Having these in mind, the rest of the playbook is pretty clear:

 resize.yml

- hosts: DOZEN-VM
  vars:
    dev: /dev/sda
    vg_name: centos
    lv_name: root
  remote_user: ansible
  tasks:
  - name: Get partition information
    parted:
      device: "{{dev}}"
    register: partinfo
    become: yes
  - debug: var=partinfo.partitions[-1].end
  - name: "Create a new partition after {{partinfo.partitions[-1].end}}KiB"
    parted:
      part_start: "{{partinfo.partitions[1].end}}KiB"
      device: "{{dev}}"
      number: "{{partinfo.partitions[-1].num + 1}}"
      flags: [ lvm ]
      state: present
    become: yes
  - name: Probe kernel about partition table changes
    command: partprobe
    become: yes
  - name: Add the partition to the volume group
    lvg:
      vg: "{{vg_name}}"
      pvs: "{{dev}}{{partinfo.partitions[-1].num}},{{dev}}{{partinfo.partitions[-1].num + 1}}"
    become: yes
  - name: Extend LVM to the whole volume group
    lvol:
      vg: "{{vg_name}}"
      lv: "{{lv_name}}"
      size: +100%FREE
    become: yes
  - name: Extend the filesystem
    filesystem:
      fstype: xfs
      dev: "/dev/mapper/{{vg_name}}-{{lv_name}}"
      resizefs: yes
    become: yes

Running against every server

We have already test connectivity to all servers, so we only need our inventory and the playbook file:

$ ansible-playbook -i inventory resize.yml

The procedure needs some seconds to complete. We have already open the console of centos7_12 in order to confirm the result:

As long as we have built our environments with some standards using orchestration methods, procedures like this have great value. We can manage noumerous servers and services with the same efford. Our environments can scale easily and be more portable as we can re-build them on any cloud provider instead of Virtualbox with minimum changes.