Server protection with Fail2ban

16 min readOct 26, 2020

As soon as you start your server, it immediately jumps into the ocean of botnets trying to login with common credentials. And if your server has anything useful, sooner or later it may become a target for script kiddies or even more advanced folks with powerful computers and latest technologies, including Artificial Intelligence.

You can save your server from the botnets and somewhat complicate life for hackers using Fail2ban.

Content

1. What is Fail2ban
2. How to install
3. Basic configuration
4. Advanced configuration
- Protecting other services
- Adding custom regexes
5. Notification with mail
6. Reporting bad IPs
7. Unban banned IPs manually
8. Reset fail2ban logs and database
9. How to uninstall fail2ban

Let’s go…

1. What is Fail2ban (F2B)

Fail2Ban is an IPF — Intrusion Prevention Framework that protects servers from external attacks.

It was created by Cyril Jaquier (Switzerland) back in 2004 and since then it is under active development by the growing community.

F2B operates by monitoring log files, e.g. auth.log, apache or nginx access logs, and taking actions on IPs with too many unwanted activities within a predefined amount of time.

Possible actions include:
- updating firewall rules,
- sending e-mail notifications,
- reporting bad IPs to a special services,
- any other action that can be done with a Python script.

Standard configuration already includes filters for various services, e.g. sshd, postfix, and many others. Every filter is designed to identify failures for that specific service through the use of complex regular expressions. It defines these regular expression patterns into a variable called failregex. A combination of a filter and predefined actions is known as a “jail”.

Speaking of F2B weaknesses, you have to know that it doesn’t work very well against a distributed brute-force attacks. The reason being that monitored log files are rescanned approximately once per second. This should be fine in most cases, yet it is possible to get more login failures than specified in “maxretry” parameter.

Here I have to mention optional but recommended software — Gamin, the File Alteration Monitor. If Gamin is installed and backend in jail.conf is set to auto or gamin, then active polling of log files is no longer required. Actions would be triggered as soon as log files are modified.

F2B system requirements are quite low. When configured with sshd and recidive jails only, F2B needs about 500 Mb of memory and loads 1 core CPU for less than 0,2% on average.

Links:
- F2B main page
- F2B GitHub repository

If you want to help with the development of F2B and / or get deeper understanding of the filters, then you are most welcome to the:
- F2B developers documentation

2. How to install

First of all, perform standard update of repos and packages with:

sudo apt update && sudo apt upgrade -y

Then you have two ways to install the software.

Easy with stable release
If you prefer faster way, system managed package and stable release, then just use this command:

sudo apt install fail2ban -y

Techie with latest release
If you prefer latest and greatest, then install from GitHub. Bear in mind that this will require a bit more of dancing. First, make sure you have Python or Python 3 installed in your system. I will use Python 3, so let’s check that you have Python 3 installed with:

python3 — version

If as a result of this command you got message: ”Command ‘python3’ not found, but can be installed with…”, then:

sudo apt install python3
sudo apt install python3-setuptools

And, just in case:

sudo apt install git

Then download the release from the official F2B repository:

cd ~
git clone https://github.com/fail2ban/fail2ban.git
cd fail2ban
sudo python3 setup.py install

Hopefully you got no errors, but if you did, please, google for solution as it is outside this post’s scope.

This next command will allow you to work more comfortably and to auto-complete F2B related commands, such as fail2ban-client, fail2ban-server, fail2ban-regex by pressing tab in the console:

sudo cp files/bash-completion /etc/bash_completion.d/fail2ban

If there is a lot happening on your server, then it is good idea to set up log rotation. After log will reach certain size, it will be archived and F2B will start to write to a fresh file. To set up F2B logs rotation:

sudo wget -O /etc/logrotate.d/fail2ban https://raw.githubusercontent.com/fail2ban/fail2ban/debian/debian/fail2ban.logrotate

Now you need to enable F2B as service.

sudo cp ~/fail2ban/build/fail2ban.service /lib/systemd/system/
cd /etc/systemd/system/
sudo ln /lib/systemd/system/fail2ban.service fail2ban.service

You shall get this message as a result:

Created symlink /etc/systemd/system/multiuser.target.wants/fail2ban.service → /etc/systemd/system/fail2ban.service.

Now let’s start F2B:

sudo systemctl start fail2ban

And check the status of the service:

sudo systemctl status fail2ban

Check version:

sudo fail2ban-client version

- Latest and stable in 2020 is version 0.10.2
- Latest and greatest in 2020 is version 1.0.1.dev1

Enable fail2ban service, so it will run after restarts:

sudo systemctl enable fail2ban.service

Also restart `rsyslog` service too:

sudo service rsyslog restart

Done.

3. Basic configuration

You configure Fail2Ban using the files in /etc/fail2ban.

Most important file is jail.conf, which I do not recommend to edit directly, since it will be re-written after every upgrade of F2B. Common practice is to make all your customizations in jail.local file. When starting, F2B reads *.conf files first and then *.local versions if they exist. Settings in the latter take over.

In all the following commands you can replace vim with your favorite editor, e.g. nano. Now, let’s create a new file, jail.local:

sudo vim /etc/fail2ban/jail.local

In this new file, to set up parameters for all jails, you define them in the section DEFAULT:

[DEFAULT]bantime = 3h
findtime = 10m
maxretry = 5

These settings will ban any IP address for three hours {bantime = 3h}, if within 10 minutes {findtime = 10m} interval, there will be detected 5 {maxretry = 5} failed connection attempts from this particular IP.

Same parameters, set under any individual jail, will have priority over general settings, allowing you to fine tune settings for each jail separately.

If you connect to your server from the static IPs all the time, it’s good idea to whitelist them, so they never get banned. You may whitelist several IPs by separating them with whitespace. Localhost is whitelisted by default.

ignoreip = 127.0.0.1 x.x.x.x y.y.y.y

Now let’s tell F2B to enable sshd jail:

[sshd]
enabled = true

Complete version of a very basic jail.local file now looks like this:

Fail2Ban LOCAL configuration file.[DEFAULT]bantime = 3h
findtime = 10m
maxretry = 5ignoreip = 127.0.0.1JAILS[sshd]
enabled = true

Time Abbreviation Formats

The time entries in F2B configuration (like findtime or bantime) can be provided as integer in seconds or as string using special abbreviation format, e. g. 600 is the same as 10m (10 minutes).

Acceptable abbreviation tokens:

years?, yea?, yy?
months?, mon?
weeks?, wee?, ww?
days?, da, dd?
hours?, hou?, hh?
minutes?, min?, mm?
seconds?, sec?, ss?

The question mark (?) follows the optional character, so to set time in days you can use days or day, da, dd or d. You can combine multiple tokens in format separated with space resp. without separator, e. g.: 1y 6mo or 1d12h30m.

Note that tokens m as well as mm means minutes, for month use abbreviation mo or mon.

You can convert time abbreviation format to seconds using fail2ban-client with option — str2sec:

> fail2ban-client — str2sec 1d12h
129600

It is also possible to configure the server using commands sent via fail2ban-client. The available commands are described in the fail2ban-client(1) manpage.

Bear in mind that fail2ban-client set only updates values for a running F2B server, so do not forget to update the configuration file accordingly if they have to be persistent!

Also see fail2ban(1) and jail.conf(5) man pages for further references:
- https://manpages.ubuntu.com/manpages/bionic/man1/fail2ban.1.html
- https://manpages.ubuntu.com/manpages/bionic/en/man5/jail.conf.5.html

4. Advanced configuration

A very simple way to increase security is to add recidive jail. Standard jail will ban and then, after a predefined amount of time, unban IP so nothing prevents this IP to repeat its’ connection attempts again. Let’s add special jail for such recidivists. Copy and paste this into your jail.local file:

[recidive]
enabled = true
bantime = 9w
findtime = 3d

Difference is that [recidive] jail points to the F2B log itself in /var/log/fail2ban.log. If the same IP appears there within the findtime amount of time (3 days here), then it gets banned for the bantime specified in this jail, 9 weeks in this case.

Now let’s set up more sophisticated banning system, which will increase ban time with every attempt by a random amount of time to complicate life for a “clever” botnets. Add this info to your jail.local:

[DEFAULT]bantime.increment = true
bantime.rndtime = 30m
bantime.maxtime = 60d
bantime.factor = 2
bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)
bantime.overalljails = true

Let’s review every parameter one by one.
- bantime.increment allows to use database for searching of previously banned IPs to increase a default ban time using special formula, which by default is banTime 1, 2, 4, 8, 16, 32…
- bantime.rndtime is the max number of seconds using for mixing with random time to prevent “clever” botnets calculating exact time after which an IP can be unbanned again.
- bantime.maxtime is the max number of seconds that ban time can reach (doesn’t grow further)
- bantime.factor is a coefficient for the formula to calculate exponential growth or common multiplier. Default value is 1 and with it ban time grows by 1, 2, 4, 8, 16…
- bantime.formula used by default to calculate next value of ban time, default is shown below. The same ban time increase will be reached by multipliers 1, 2, 4, 8, 16, 32…

Example of the default formula:

bantime.formula = ban.Time * (1<<(ban.Count if ban.Count<20 else 20)) * banFactor

More aggressive example of formula, it has the same values only with factor “2.0 / 2.885385”:

bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)

So, considering our previous settings plus these more advanced, F2B will now do the following:
- Ban any IP address for 3 hours, if it makes 5 failed connection attempts within 10 minutes interval
- Save this IP to its internal database
- Unban it after 3 hours
- If this IP will make another failed connection attempt, F2B will calculate new bantime by formula
- Ban this IP for this new, increased and random time.
- bantime will be increased randomly and exponentially with every new attempt, but not exceed 60 days.
- If this IP will be blocked 5 times within the 3 days, then it will be banned for 9 weeks (put in recidive jail).

There is a lot of space for adjusting these settings according to your specific situation, which may involve logs reviews and analysis. As they say, security is a process, not a one-time procedure.

4.1 Protecting other services

By default all jails are disabled. You need to enable jails relevant to your setup in your jail.local file by adding enabled = true under a jail name. For example, if you have Apache web server installed, you may want to add this to your jail.local:

[apache-auth]
# Ban hosts with password authentication failures
enabled = true
port = http,https
logpath = %(apache_error_log)s[apache-badbots]
# Ban spammer robots crawling the web for email addresses.
port = http,https
logpath = %(apache_access_log)s
bantime = 172800
maxretry = 1[apache-noscript]
# Ban hosts searching for scripts on the website to execute and exploit
enabled = true
port = http,https
logpath = %(apache_error_log)s
maxretry = 6[apache-overflows]
# Ban hosts attempting to request unusually long and suspicious URLs
enabled = true
port = http,https
logpath = %(apache_error_log)s
maxretry = 2[apache-nohome]
# If you do not provide access to web content within users’ home directories
enabled = true
port = http,https
logpath = %(apache_error_log)s
maxretry = 2

For exim, add

[exim]
enabled = true

and optionally

[exim-spam]
enabled = true

For postfix, add

[postfix]
enabled = true

and so on. Complete list of all pre-configured jails you will find in /etc/fail2ban/jail.conf file.

For many other services there are different solutions. For instance, WordPress has special plugin. After installing this plugin from the admin panel, you need to copy filter

sudo cp /var/www/html/wp-content/plugins/wp-fail2ban/filters.d/wordpress-hard.conf /etc/fail2ban/filter.d/

And enable this jail in jail.local:

[wordpress-hard]
enabled = true
port = http,https
filter = wordpress-hard
logpath = /var/log/auth.log
bantime = 1h
maxretry = 1

In general, if you do not have filter for your application / service, and cannot find one on the Internet, you can write your own, by adding list of regexes to /etc/fail2ban/filter.d/non-standard.conf. Then you can enable your non standard jail by adding to jail.local:

[non-standard]
enabled = true
port = port your app/service uses
logpath = where its log file is located

Now, when you know general principles of F2B, you can find, configure and adjust any settings for any service.

4.2 Adding custom regexes

As you already know, F2B watches the log(s) you told it to watch and searches there for entries using regular expressions. For instance, regexes in /etc/fail2ban/filter.d/postfix.conf for Postfix log look like this:

failregex = ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 554 5\.7\.1 .*$
 ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 450 4\.7\.1 : Helo command rejected: Host not found; from=<> to=<> proto=ESMTP helo= *$
 ^%(__prefix_line)sNOQUEUE: reject: VRFY from \S+\[<HOST>\]: 550 5\.1\.1 .*$
 ^%(__prefix_line)simproper command pipelining after \S+ from [^[]*\[<HOST>\]:?$

Yet it happens that standard filters may miss some entries in the logs, because they are unusual. So you might want to check this situation from time to time and append custom regexes to improve security. How to?

F2B has special utility for this, called fail2ban-regex, which tests regular expressions. General usage is like this:

fail2ban-regex [OPTIONS] LOG REGEX [IGNOREREGEX]

- [OPTIONS] — check complete list of options by command fail2ban-regex -h.
- LOG is a log entry string or path to a log file.
- REGEX is a string (actual regex) or path to a filter file, i.e. list of regexes.
- [IGNOREREGEX] is the same, string or path to file.

First, identify, what log and what service you want to test it for, e.g. auth.log and sshd.conf. In other words, you want to see what standard set of filters in sshd.conf misses in auth.log. Then, enter this command in the console:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf — print-all-missed > /home/user_name/missed.txt

Key here is to use option — print-all-missed. By default it will print everything to stdout, i.e. to the console, but you may want to redirect output to a text file (e.g. missed.txt) for further exploration and processing. Let’s examine the result:

This is actual output of the above command for auth.log processed by standard sshd.conf AND custom set of regexes in sshd.local (slightly edited to remove sensitive info). Even in this case filters missed, as you may see, some entries in the log file. Let’s have a closer look at important points, marked with red arrows:

1. There are 812 entries in auth.log that matched failregexes. Number in square brackets at the beginning of every regex shows how many times it worked.
2. There are 2959 entries that matched ignoreregex, i.e. entries I told system to ignore.
3. And finally, as you may see, 35 lines were missed. Which means that IPs 146.52.214.41 and 37.135.222.203 skipped further processing by F2B.

How come? Well, missed lines appear due to no regex or poorly written regex. So you need to make / test your custom regex that will catch missed lines and then add them to /etc/fail2ban/filter.d/sshd.local. For further details I do recommend to study appropriate section of the developers documentation.

This is truncated version of my /etc/fail2ban/filter.d/sshd.local file:

# Fail2Ban filter for openssh — local additions[Definition]cmnfailre = ^Bad protocol version identification .* from <HOST>%(__on_port_opt)s$
 ^Connection closed by ((authenticating|invalid)? user \S+ )?<HOST>%(__on_port_opt)s%(__suff)s$
 ^Connection reset by <HOST>%(__on_port_opt)s%(__suff)s$
 ^Did not receive identification string from <HOST>%(__on_port_opt)s$
 ^Disconnected from ((?:authenticating|invalid) user \S+ )?<HOST>%(__on_port_opt)s%(__suff)s$
 ^Invalid user \S* from <HOST>%(__on_port_opt)s$
 ^message repeated \d+ times: \[ Failed password for root from <HOST>%(__on_port_opt)s ssh2\]$
 ^pam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=<F-USER>\S*</F-USER>\
 ^PAM \d more authentication failure(s)?; logname= uid=0 euid=0 tty=ssh ruser= rhost=<HOST>( user=root)?$
 ^Received disconnect from <HOST>%(__on_port_opt)s:.*\s%(__suff)s$
 ^Unable to negotiate with <HOST>%(__on_port_opt)s:\s.*\s%(__suff)s$
 ^User \S+ from <HOST> not allowed because not listed in AllowUsers$
 ^Disconnecting invalid user \S* <HOST>%(__on_port_opt)s:
 ^(error: )?maximum authentication attempts exceeded for .* from <HOST>(?: port \d*)?(?: ssh\d*)? \[preauth\]$ignoreregex = pam_unix\((cron|sshd|sudo|systemd-user):session\): session (open|clos)ed for user (root|USER_NAME)( by \(uid=0\))?$
 pam_unix\(sudo:session\): session opened for user root by USER_NAME\(uid=0\)$
 pam_unix\(sshd:auth\): check pass\; user unknown$
 New session \d+ of user USER_NAME\.$
 Removed session \d+\.$
 User child is on pid \d+$
 Timeout, client not responding\.$
 \s+USER_NAME \: TTY=pts\/\d \; PWD=\/home\/USER_NAME(\/.*)? \; USER=root \; COMMAND=.*$
 Disconnected from user USER_NAME <HOST>%(__on_port_opt)s$

If you want to copy it to your server, replace USER_NAME in the text above with your real user name.

I have also added crontab job, which runs every day at 08:00 with:

/usr/local/bin/fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf — print-all-missed > /home/USER_NAME/missed.txt

and then simply cat missed.txt to check if anything outstanding was missed by F2B sshd filters.

You can also check other logs and other filters this way and add your regexes.

5. Notification with mail

You can enable email notifications if you wish to receive email whenever a ban takes place. To do so, you will have to first set up an MTA (Mail Transfer Agent) on your server so that it can send out emails. To learn how to use Postfix for this task, follow this guide. Feel free to use any other MTA and change settings below accordingly.

Once you have your MTA set up, you need to adjust settings within the DEFAULT section of the jail.local. Start by setting the mta directive. If you set up Postfix, like the above tutorial demonstrates, set this value:

[DEFAULT]
mta = postfix

Now you need to add emails and some other fields:
- destemail: The email address where you would like to receive the emails.
- sender: The email address from which F2B will send emails (see the guide above).
- sendername: The text in the “Sender” field of the notification emails.

[DEFAULT]
mta = postfix
destemail = youraccount@email.com
sender = user@localhost.localdomain
sendername = Fail2BanAlert

ACTIONS

Now you need to adjust the action setting. Just as a quick reminder, in F2B parlance, an “action” is the procedure followed when a client fails too many times.

- The default %(action_)s only bans a host.
- %(action_mw)s will ban and send an email with a WHOIS report on the offending IP address.
- %(action_mwl)s will ban, send an email with the WHOIS report and all relevant lines in the log file.

Select level of detalization that suits you better and set it up:

[DEFAULT]
action = %(action_mwl)s

This can also be changed on a jail-specific basis (under a jail name), so that you’ll get different details for different jails.

To conclude this chapter, here is enlightening story of Ken Tossel related to the email notifications.

“…until July 2011, Fail2Ban’s default configuration file specified the From: address fail2ban@mail.com and the To: address you@mail.com, which are both hosted by freemail provider Mail.com. I guess the person who set this up assumed that mail.com was invalid. you@mail.com is either reserved or filled to the limit, so when Fail2Ban detects an attack on a misconfigured server, its report gets bounced back to the sender, fail2ban@mail.com. I registered that address back in 2010…”

In July 2014, Ken had parsed all 500,000 reports and made the charts that you can see on his site: https://ken.tossell.net/ftb/

6. Reporting bad IPs

Shall you want to support the community by reporting offensive IPs, so that others can benefit from this information, there are several services that allow you to do so. Let’s review one in more details and others will be mentioned at the end of this section.

AbuseIPDB

- Step 1 — Create AbuseIPDB account
- Step 2 — Create AbuseIPDB API key

To make F2B report IPs, add commands below to your jail.local file. Add these actions to your jails individually so you can customize the AbuseIPDB report categories and make your reports more specific. For instance, to report IPs for sshd brute forcing, add this under [sshd] jail:

[sshd]
action_abuseipdb = abuseipdb
action = %(action_)s
 %(action_abuseipdb)s[abuseipdb_apikey=”YOUR.API.KEY”, abuseipdb_category=”18,22"

Most popular AbuseIPDB report categories:

| Name | # |
| — — — — — — — — -|: — |
| FTP Brute-Force | 5 |
| Port Scan |14 |
| Hacking |15 |
| Brute-Force |18 |
| Bad Web Bot |19 |
| SSH |22 |
| Web App Attack |21 |

Full list of all categories is here: https://www.abuseipdb.com/categories

Optionally, you can customize the message by editing /etc/fail2ban/action.d/abuseipdb.conf:

actionban = curl — tlsv1.0 — fail ‘https://api.abuseipdb.com/api/v2/report' \
 -H ‘Accept: application/json’ \
 -H ‘Key: <abuseipdb_apikey>’ \
 — data-urlencode ‘ip=<ip>’ \
 — data-urlencode ‘comment=F2B blocked SSH BF’ \
 — data ‘categories=<abuseipdb_category>’

Then your reports on AbuseIPDB site will look like this:

You can replace F2B blocked SSH BF comment with your own.

More information and troubleshooting is on the official site: https://www.abuseipdb.com/fail2ban.html

Other online services to report IPs

Blocklist.de
BadIPs. Setup F2B by following their official documentation.

7. Unban banned IPs manually

This is very simple command:

sudo fail2ban-client set JAIL_NAME unbanip IP_ADDRESS

If IP_ADDRESS is not blocked by this particular jail, you will see 0 as output, otherwise 1.

Couple of times I managed to get my own IP blocked :) working from various geographical locations. In this case, simply reconnect to your server through any proxy and unblock your current IP with the command above.

8. Reset Fail2ban logs and database

All banned IP addresses are saved in this file:
/var/lib/fail2ban/fail2ban.sqlite3.

Even if you clear all logs, F2B will recreate all banned IPs out of the database after every service restart until the bantime for the IP runs out. To clear the whole F2B log and every banned IP from the database do the following steps. It’s a good method to reset F2B to point zero resetting every banned IP.

sudo fail2ban-client stop
sudo truncate -s 0 /var/log/fail2ban.log
sudo rm /var/lib/fail2ban/fail2ban.sqlite3
sudo fail2ban-client restart

Don’t worry — F2B recreates the fail2ban.sqlite3 file after restarting the service

You can also manually delete the IP from the database without loosing all other banned IPs and if you don’t want to use whitelisting:

1. Download the DB Browser for SQLite from http://sqlitebrowser.org/
2. Install the DB Browser on your local computer
3. Download the fail2ban.sqlite3 file from /var/lib/fail2ban/fail2ban.sqlite3
4. Start the DB Browser and open your fail2ban.sqlite3 file
5. Search the database for the banned IP
6. Select it and click on “delete row”
7. Save the fail2ban.sqlite3 file and close the program
8. Connect to your server with ssh and type service fail2ban stop
9. Type truncate -s 0 /var/log/fail2ban.log
10. Upload edited fail2ban.sqlite3 back to /var/lib/fail2ban/fail2ban.sqlite3
11. Turn back to your ssh connection and type service fail2ban restart

One more option is to edit this database directly on server with sqlite3 command line utility.
https://manpages.ubuntu.com/manpages/bionic/en/man1/sqlite3.1.html

9. How to uninstall Fail2ban

To remove just F2B package, stop the service and remove package:

sudo fail2ban-client stop
sudo apt remove fail2ban

To remove the F2B package and any other dependent packages:

sudo apt remove — auto-remove fail2ban

If you also want to delete configuration and/or data files of F2B:

sudo apt purge fail2ban

To delete configuration and/or data files of F2B and it’s dependencies:

sudo apt purge — auto-remove fail2ban

Conclusion

Setting up F2B to protect your server is fairly straight forward in the simplest cases. However, F2B provides a great deal of flexibility to construct policies that will suit your specific security needs. You can find many pieces to tweak and change as your needs will evolve. Learning the basics of how to protect your server with F2B can provide you with a great deal of security with minimal effort.

I hope that this information, based on my notes, carefully collected and tested in practice for many years, will help you to start / configure the protection of your server.