Running Linux/UNIX servers for several sites I was fed up with seeing the numerous 403 and 404 errors being reported daily in Apache’s access log files which had been growing in number over the past several years. So, I decided to make a stance and to start sticking two fingers up to constant Brute Force Attacks.
Corrupting and defacing my logfiles with your ‘S***’  NO MORE!

Following detailed analysis of this type of data it became apparent that there was no common source of traffic but the majority of file requests did have pattern matches. It was fruitless to block IP address after IP address because I would need a Super computer to sieve through the endless list of blocked addresses before presenting my sites’ data to legitimate visitors, probably many seconds later.

So, what did I do to block all malicious traffic without manually entering even one IP address anywhere?
Well, I went on the hunt using good old Google to scour Linux/UNIX security web forums to find the Crème de la crème of intrusion prevention software.

What did I find? Well, the Crème de la crème by a long way turned out to be a Python based program called fail2ban. It is able to run on POSIX systems that have an interface to a packet-control system or firewall installed locally, for example, iptables or TCP Wrapper.
Fail2ban is primarily focused on SSH attacks, although it can be configured to work for any applications that use log files. Yes, you read correctly, any application that’s on a server that communicates to the outside of it’s RJ45 socket or wireless antennae.

I only immediately needed the application to address my 403 & 404 errors, so here is what I did.

fail2ban was and still is part of my Linux distro, so I installed it via the software management utility. OK, that’s the easy part completed.

The fail2ban documentation states that no changes should be carried out to the ‘jail.conf’ file, but to the ‘jail.local’ file instead.

Now for the configuration:

In my case, the majority of files were installed in ‘/etc/fail2ban’, so off I trot and set the ‘jail.conf’ to match the requirements of my server. The ‘jail.local’ file is read after the jail’conf’ file and any settings present will override those of the latter. If you feel confident enough to edit ‘jail.conf’, here’s the basic changes required. (after making a safe copy of it).

First, I set my whitelisted IP addresses for my allowed local machines so that in the event of accidentally blocking myself, I wouldn’t be, if you know what I mean.

Tones: # vi jail.conf
........
# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
127.0.0.1/8 ::1 192.168.1.12 192.168.1.35 192.168.10.0/24

The settings above are global, so will cover all jails that you setup. Note: As stated a space is required between each address.

If you wish to have different whitelisted addresses for different jails then do not edit the ‘ignoreip’ in ‘jail.conf’, but manually add IP addresses for the respective JAIL -NAME by:

Tones: # fail2ban-client set JAIL-NAME addignoreip 192.168.1.35 googlebot.com
Tones: # fail2ban-client set JAIL-NAME2 addignoreip 192.168.1.12 googlebot.com

Set the email address section.

destemail: The email address where you would like to receive the emails.
sender: The email address from which Fail2ban will send emails.

That should be enough to get you going initially. For further changes all documentation can be found here: https://www.fail2ban.org/wiki/index.php/Main_Page

All application preconfigured jails can be found in ‘/etc/fail2ban/filter.d’ directory. I could not find a preset filter for my requirements so went about and configured two of my own in the ‘filter.d’ directory. I created my required 403-error and 404-error files which include the headers required to be read from my Apache access logfiles. Instead of writing a filter for each additional hosted site on my server, one entry was sufficient for all access logfiles. So here are the regex entries.

Tones: # cd /etc/fail2ban/filter.d
Tones: # cat 403-error.conf
[Definition]
failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*" 403
ignoreregex =

Tones: # cat 404-error.conf
[Definition]
failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*" 404
ignoreregex =

I then created my rules for each error in ‘/etc/fail2ban/jail.local’

Tones: # cat jail.local

[404-error]

enabled = true
port = http,https
filter = 404-error
logpath = /var/log/apache2/*access.log
maxretry = 3
findtime = 21600
bantime = 604800

#---------------------------------------------------------
[403-error]

enabled = true
port = http,https
filter = 403-error
logpath = /var/log/apache2/*access.log
maxretry = 2
findtime = 21600
bantime = 604800

#---------------------------------------------------------

Fine tune the maxretry, findtime and bantime to suit your needs.

bantime: The length of time in seconds for which an IP is banned. If set to a negative number, the ban will be permanent. The default value of 600 is set to ban an IP for a 10-minute duration.
findtime: The length of time between login attempts before a ban is set. For example, if Fail2ban is set to ban an IP after five (5) failed log-in attempts, those 5 attempts must occur within the set 10-minute findtime limit. The findtime value should be a set number of seconds.
maxretry: How many attempts can be made to access the server from a single IP before a ban is imposed. The default is set to 3.

Enable ‘fail2ban’ and set it to automatically start upon every startup of the system.

In my case I initially started it by:

Tones: # service fail2ban start

This created a logfile in /var/log named ‘fail2ban.log’

If all has gone well you should see something like the below with no errors.

2019-02-15 16:52:12,353 fail2ban.server [25970]: INFO Starting Fail2ban v0.10.3.fix1
2019-02-15 16:52:12,359 fail2ban.database [25970]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2019-02-15 16:52:12,360 fail2ban.jail [25970]: INFO Creating new jail '404-error'
2019-02-15 16:52:12,370 fail2ban.jail [25970]: INFO Jail '404-error' uses pyinotify {}
2019-02-15 16:52:12,374 fail2ban.jail [25970]: INFO Initiated 'pyinotify' backend
2019-02-15 16:52:12,376 fail2ban.filter [25970]: INFO Added logfile: '/var/log/apache2/d-access.log' (pos = 27263, hash = b53c16d4afd3ebaa9abb6f5c64a5033c)
2019-02-15 16:52:12,377 fail2ban.filter [25970]: INFO Added logfile: '/var/log/apache2/access.log' (pos = 2707349, hash = 4ca5eaa80af0e4c9d28b8c534bfb3dad)
2019-02-15 16:52:12,377 fail2ban.filter [25970]: INFO maxRetry: 3
2019-02-15 16:52:12,379 fail2ban.filter [25970]: INFO encoding: UTF-8
2019-02-15 16:52:12,379 fail2ban.actions [25970]: INFO banTime: 604800
2019-02-15 16:52:12,380 fail2ban.filter [25970]: INFO findtime: 21600
2019-02-15 16:52:12,381 fail2ban.jail [25970]: INFO Creating new jail '403-error'
2019-02-15 16:52:12,381 fail2ban.jail [25970]: INFO Jail '403-error' uses pyinotify {}
2019-02-15 16:52:12,384 fail2ban.jail [25970]: INFO Initiated 'pyinotify' backend
2019-02-15 16:52:12,386 fail2ban.filter [25970]: INFO Added logfile: '/var/log/apache2/d-access.log' (pos = 27263, hash = b53c16d4afd3ebaa9abb6f5c64a5033c)
2019-02-15 16:52:12,387 fail2ban.filter [25970]: INFO Added logfile: '/var/log/apache2/access.log' (pos = 2707349, hash = 4ca5eaa80af0e4c9d28b8c534bfb3dad)
2019-02-15 16:52:12,388 fail2ban.filter [25970]: INFO maxRetry: 2
2019-02-15 16:52:12,389 fail2ban.filter [25970]: INFO encoding: UTF-8
2019-02-15 16:52:12,389 fail2ban.actions [25970]: INFO banTime: 604800
2019-02-15 16:52:12,390 fail2ban.filter [25970]: INFO findtime: 21600
2019-02-15 16:52:12,392 fail2ban.jail [25970]: INFO Jail '404-error' started
2019-02-15 16:52:12,393 fail2ban.jail [25970]: INFO Jail '403-error' started

Now, every time any of my sites receive a 403 or 404 error the culprit IP address will be added to the fail2ban database and shown in the ‘fail2ban’ logfile. In the example above 404-error, if an IP hits 404 more than three times in six hours it will be banned for one week.

To check for blocked IP addresses in your firewall’s ‘iptables’ do the following:

Tones: # fail2ban-client status 404-error
Status for the jail: 404-error
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/apache2/access.log /var/log/apache2/d-access.log
`- Actions
   |- Currently banned: 1
   |- Total banned:     1
   `- Banned IP list:   212.72.153.226

In the 404-error jail above we have captured one IP for a total of one week when it will automatically be released without charge.

Serves the ‘Bugger’ right.

Cheers

Tone