Distributing fail2ban actions

Fail2ban

Fail2ban is a service that parses log files and can perform configured actions when a given regex is found. Usually it is used to ban offending IP addresses using firewall rules on linux machines. In situations that we have more than one server under our responsibility, it could be better to apply those actions not only to the server experienced the offending behaviour. For example we may refuse connections to our web server from the IP that just messed up with our mail server and so on.

For this demonstration we will configure only one jail and one action.

Jail configuration (template):

This is a simple ssh jail that parses /var/log/secure.log and detects failed logins:

 jail.local.j2

[DEFAULT]
# Ban hosts for one hour:
bantime = 3600

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = redis-publisher

[sshd]
enabled = true

Action configuration (template):

For the action configuration we will use the following file. It is a copy from the default firewallcmd-ipset.conf with the variables actionban and actionunban commented out and changed to call a python script with a JSON string as argument.

 redis-publisher.conf.j2

[INCLUDES]
  
before = iptables-common.conf

[Definition]

actionstart = ipset create fail2ban-<name> hash:ip timeout <bantime>
              firewall-cmd --direct --add-rule ipv4 filter <chain> 0 -p <protocol> -m multiport --dports <port> -m set --match-set fail2ban-<name> src -j <blocktype>

actionstop = firewall-cmd --direct --remove-rule ipv4 filter <chain> 0 -p <protocol> -m multiport --dports <port> -m set --match-set fail2ban-<name> src -j <blocktype>
             ipset flush fail2ban-<name>
             ipset destroy fail2ban-<name>

# actionban = ipset add fail2ban-<name> <ip> timeout <bantime> -exist
actionban = /usr/sbin/publisher.py "{\"action\": \"ban\", \"ip\": \"<ip>\", \"bantime\": \"<bantime>\", \"name\": \"<name>\", \"published_by\": \"\"}"

# actionunban = ipset del fail2ban-<name> <ip> -exist
actionunban = /usr/sbin/publisher.py "{\"action\": \"unban\", \"ip\": \"<ip>\", \"bantime\": \"\", \"name\": \"<name>\", \"published_by\": \"\"}"

[Init]

chain = INPUT_direct
bantime = 600

Redis

Redis is an open source key-value data store, used as a database, cache and message broker. Since redis is using memory for data storing it is extremely fast. We can use it for our little project as message broker in Pub/Sub (publish/subscribe) mode to distribute messages from any of our servers to all of them. You can use almost any programming languge to integrate with redis. We will use python and the recommended "redis-py" library for this demo.

About configuration, we leave everything to default changing only the binding ip address (by default redis listens only on localhost).

Redis configuration (template):

 redis.conf.j2

bind {{ ansible_default_ipv4.address }}
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile /var/log/redis/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

Ansible

Ansible is the most known orchestration - automation tool that we honored here with multiple articles. It is simple, agentless and works! You need to be familiar with ansible to follow this demo.

We will write an ansible-playbook that we can run against groups of servers (eg. servers that are exposed to the web). If a new server is added later, we can add it's hostname in our ansible-inventory and run the same playbook again.

A simple inventory file with the groups needed for this demo is the following:

 hosts

[REDIS-CLIENTS]
aphrodite
artemis
athena

[REDIS-SERVER]
aphrodite

Note that redis server "aphrodite" is also a redis client.

The ansible playbook need to:

• Install all required packages depending on whether host is running redis-server or just client.
• Configure all the services handling restarts.
• Secure the environment configuring firewall and selinux (if we are enforcing selinux policys).

For now, let us finish with prerequisites.

What else we need?

Excluding the above, we need two scripts:

1. The first one can connect to redis server, publish a new message and exit. Of course this will be triggered by fail2ban when offensive behavour is detected.
2. The second will be run as linux service, listen for messages and perform system actions.

A simple "publisher" python script is the following (template). It is just publishing its first argument on execution (sys.argv[1]).

 publisher.py.j2

#!/usr/bin/env python3
import sys
import redis


redis_server = "{{ groups['REDIS-SERVER'][0] }}"
redis_port = 6379
redis_db = 0
redis_channel = 'fail2ban'


def encryptor(message):
    # Out of article's scope
    return message


def main():
    message = encryptor(sys.argv[1])
    r = redis.StrictRedis(
        host=redis_server,
        port=redis_port,
        db=redis_db
        )
    p = r.pubsub()
    r.publish(redis_channel, message)
    p.close()


main()

The script for redis listener is a little more complex. It gets the JSON message from redis turning it to a python dictionary object and executes the same command that fail2ban would execute locally. We also added a basic logger that is writing in /var/log/fail2ban.log file.

 redis_listener.py.j2

#!/usr/bin/env python3
import redis
import logging
import json
import subprocess


redis_server = "{{ groups['REDIS-SERVER'][0] }}"
redis_port = 6379
redis_db = 0
redis_channel = 'fail2ban'
logfile = '/var/log/fail2ban.log'

r = redis.StrictRedis(host=redis_server, port=redis_port, db=redis_db)
p = r.pubsub()
p.subscribe(redis_channel)
logging.basicConfig(
    format='%(asctime)s %(levelname)s: %(message)s',
    filename=logfile,
    level=logging.DEBUG
    )


def decryptor(message):
    # Out of article's scope
    return message


def execute(message):
    if message['action'] == 'ban':
        command = 'ipset add fail2ban-{name} {ip} timeout \
        {bantime} -exist'.format(**message)
    else:
        command = 'ipset del fail2ban-{name} {ip} -exist'.format(**message)
    logging.debug(
        '{host} triggered: {command}'.format(
            host=message['published_by'],
            command=command
            ))
    return_code = subprocess.call(command, shell=True)
    if not return_code == 0:
        logging.info('Command: {command} failed with rc={rc}'.format(
            command=command,
            rc=return_code
            ))
    else:
        logging.debug('Command: {command} executed succesfully'.format(
            command=command
            ))


def main():
    for record in p.listen():
        if record['type'] == 'message':
            message = json.loads(decryptor(record)['data'].decode('utf-8'))
            execute(message)


main()

Comment/Disclaimer:

This article is basically a proof of concept. If you intend to use it on a production environment, you need to be serious about message validation and encryption. The script "as is" could allow any user of your servers inject shell commands as redis messages and run them as root on all your servers.

Since the script will run as linux service we also need a simple systemd unit file:

 redis_listener.service

[Unit]
Description=Redis Listener: Daemon that listen, ban and unban
Before=network-pre.target
Wants=network-pre.target
After=polkit.service
Conflicts=iptables.service ip6tables.service ebtables.service ipset.service

[Service]
ExecStart=/usr/sbin/redis_listener.py --nofork --nopid
ExecStop=/bin/kill -HUP $MAINPID
# supress to log debug and error output also to /var/log/messages
StandardOutput=null
StandardError=null
Type=idle

[Install]
WantedBy=multi-user.target

The playbook

 fail2ban-distributed.yml

- hosts: REDIS-CLIENTS
  vars:
  remote_user: ansible
  tasks:
  - name: Upgrade all packages
    yum:
      name: '*'
      state: latest
    become: yes
  - name: Install epel repository
    yum:
      name: 'epel-release'
      state: latest
    become: yes
  - name: Install fail2ban, redis, python3, pip
    yum:
      name: '{{ item }}'
      state: latest
    with_items:
     - fail2ban
     - redis
     - python34-pip
     - python34
    become: yes
    notify:
    - restart fail2ban
    - restart redis
  - name: using pip to install python libraries
    pip:
      name: redis
      executable: pip3
    become: yes
  - name: Configure redis
    template:
      src: redis.conf.j2
      dest: /etc/redis.conf
      owner: redis
      group: root
      mode: 0640
    when: "'REDIS-SERVER' in group_names"
    become: yes
    notify:
    - restart redis
  - name: Enable redis port on redis server only for redis clients
    firewalld:
      zone: public
      rich_rule: rule family=ipv4 source address={{ hostvars[item]['ansible_enp0s3']['ipv4']['address'] }} port protocol=tcp port=6379 accept
      permanent: true
      state: enabled
    with_items: "{{ groups['REDIS-CLIENTS'] }}"
    when: "'REDIS-SERVER' in group_names"
    become: yes
    notify:
    - restart firewall
  - name: Install publisher python script
    template:
      src: publisher.py.j2
      dest: /usr/sbin/publisher.py
      owner: root
      group: root
      mode: 0700
    become: yes
    notify:
    - restart fail2ban
  - name: Install fail2ban action conf
    template:
      src: redis-publisher.conf.j2
      dest: /etc/fail2ban/action.d/redis-publisher.conf
      owner: root
      group: root
      mode: 0644
    become: yes
    notify:
    - restart fail2ban
  - name: Configure fail2ban SSH jail
    template:
      src: jail.local.j2
      dest: /etc/fail2ban/jail.local
      owner: root
      group: root
      mode: 0644
    become: yes
    notify:
    - restart fail2ban
  - name: Copy selinux policy file
    copy:
      src: my-python3.pp
      dest: /root/my-python3.pp
      owner: root
      group: root
      mode: 0644
    become: yes
  - name: Copy unit file to systemd
    copy:
      src: redis_listener.service
      dest: /etc/systemd/system/redis_listener.service
      owner: root
      group: root
      mode: 0755
    become: yes
    notify:
    - restart redis_listener
  - name: Install redis_listener
    template:
      src: redis_listener.py.j2
      dest: /usr/sbin/redis_listener.py
      owner: root
      group: root
      mode: 0700
    become: yes
    notify:
    - restart redis_listener
  - name: install selinux policy file
    command: semodule -i /root/my-python3.pp
    become: yes
  handlers:
    - name: restart fail2ban
      systemd:
        name: fail2ban.service
        state: restarted
        enabled: True
      become: yes
    - name: restart redis
      systemd:
        name: redis.service
        state: restarted
        enabled: True
      become: yes
      when: "'REDIS-SERVER' in group_names"
    - name: restart firewall
      systemd:
        name: firewalld.service
        state: restarted
        enabled: True
      become: yes
      when: "'REDIS-SERVER' in group_names"
    - name: restart redis_listener
      systemd:
        name: redis_listener.service
        state: restarted
        enabled: True
      become: yes

Comments:

1. The playbook was tested on CentOS Linux release 7.4.1708 servers with ansible 2.4.2.0.
2. We tried playbook task names to be descriptive.
3. Network interface names are hardcoded in the playbook (enp0s3 - the same for all servers). If your environment is less standardized you can configure interfaces as host variables in your inventory.
4. On firewall configuration (module firewalld) check how we are looping over [REDIS-CLIENTS] group and running only on [REDIS-SERVER].
5. If you are running selinux with "enforcing" policy you will find out that connection to redis server will be refused on clients. To overcome this, you need to generate and apply a new selinux policy (check the quote in the end of the article). In playbook, my-python3.pp file mentioned is generated manually once and installed to all clients.

Distributed fail2ban in action

The following lines are copied from /var/log/fail2ban.log file from all servers after some failed logins from ip 192.168.16.49 to server aphrodite:

aphrodite:

artemis:

athena:

... an hour later:

aphrodite:

artemis:

athena:

To generate and install a new selinux policy file, follow the steps:

1. Install "setroubleshoot-server" package (yum install -y setroubleshoot-server)
2. After a denial from selinux (check /var/log/audit/audit.log file) run sealert -a /var/log/audit/audit.log
3. This will analyze audit log suggesting actions