What is fail2ban and why you should install it.
In this day and age every web site owner or administrator is obsessed with improving the amount of traffic that flows to their site, even though little thought is given to weather or not, that traffic is of any use to their site or what harm it is potentially doing.
Read why you need to consider who is coming to your site and what you can do to protect yourself from spammers and scammers.
Putting fail2ban in context.
If you owned a corner store and there was a local school with 2000 students just up the street, you would want to attract the school kids to your store because they could be a good source of revenue. If each student spends $5 in your store that would increase your turn over by $10,000 each day.
Wow! what a corner store you would have. You are going to be rich in know time. In this scenario your site is the corner store and the school students are the visitors or traffic to your site.
But then the reality starts to sink in. 2000 kids swarm into your store every day after school. There is no room to move. They are all carrying their school bags pushing and shoving, yelling and arguing with each other.
Fights break out and stock gets knocked off your shelves and broken. In the confusion little people, being people see an opportunity and stock starts to disappear in to school bags and walk out the door with out being paid for. If that continues you will be broke in know time. Similarly at your site hackers and bad bots will try to run a muck and do as much damage as they can.
So at you corner store you employ a rent-a-cop who makes the school kids enter the store in smaller groups, leave there bags outside and any who fight or try to shoplift get banned. You don’t make quite as much as you thought you would in the first instance but you don’t go broke either. You have to look at the traffic to your server in the same way. So you will need a rent-a-cop or in this case more of a download-a-cop, and that is where fail2ban comes in. Fail2ban is a small open source program that watches your log files and tells your firewall to prevent misbehaving IP addresses from accessing your server.
Installing fail2ban.
Fail2ban can be downloaded from http://www.fail2ban.org/wiki/index.php/Downloads or if you use Ubuntu for your server you can go to the command line and enter
apt-get install fail2ban
and then fail2ban will be installed and started automatically for you. Many of the other linux distributions have there own way of installing fail2ban. So if you are not using ubuntu try the package manager on your operating system. There is also a version of fail2ban for windows servers.
Just like the rent-a-cop you will have to give fail2ban some instructions to tell it what to look for, what to ban, when and for how long. These instructions come in 3 parts which I will cover in detail later. For now install fail2ban, because fail2ban works for SSH and Apache as soon as you install and start it. So stop reading this article and install fail2ban and then return and we can continue.
The importance of understanding your log files.
Before we get into how to configure fail2ban any further, you need to understand a bit about your log files. Logs are a record of, actions that have been taken, by your server. The log is written after the action has already happened.
This is important to understand because fail2ban will NOT and can NOT ban a hack attempt before it happens or while it is in progress. However there is a fine line between the terms “before”, “while” and “after”.
What you need to understand is that someone or some script trying to hack your server will almost never just try once and then go away. If someone just makes one attempt at something and then they go away they are not really a concern. Usually though they will try repeatedly until they get in, get board, or get frustrated and go away. But more often though they will come back again and again and again…
Examining log files in detail.
Lets look at your ssh log file for example. Each time someone tries to access your server using ssh they have to enter a user name and password combination to gain access. When they hit enter the server tries to authenticate the information provided, weather or not the attempt is successful a log entry is written. If the attempt is by a hacker what you will see in the logs is something like the log entries listed below.
ssh
May 27 09:58:16 yourserver sshd[11785]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=210.51.52.132 user=root
May 27 09:58:18 yourserver sshd[11785]: Failed password for root from 210.51.52.132 port 52191 ssh2
May 27 09:58:21 yourserver sshd[11787]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=210.51.52.132 user=root
May 27 09:58:23 yourserver sshd[11787]: Failed password for root from 210.51.52.132 port 52596 ssh2
May 27 09:58:26 yourserver sshd[11790]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=210.51.52.132 user=root
May 27 09:58:27 yourserver sshd[11790]: Failed password for root from 210.51.52.132 port 52824 ssh2
May 27 09:58:30 yourserver sshd[11792]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=210.51.52.132 user=root
May 27 09:58:32 yourserver sshd[11792]: Failed password for root from 210.51.52.132 port 53029 ssh2
Above you can see 4 separate hack attempts. Each attempt starts with the date, your server name and the program writing to the log, in this case sshd “May 27 09:58:16 yourserver sshd”. Not all logs follow this pattern exactly, but most are very similar. In the case of ssh there are two lines per log entry and each line begins with the date.
The real clue to this section of the logs reporting hack attempt are the phrases “authentication failure” and “Failed password for root”. All importantly sshd reports that the IP Address is “210.51.52.132” which is the IP Address where the hack attempt is coming from. In other words the IP Address is the numbers assigned to the hackers internet connection. With that information in hand fail2ban can tell Iptables or Shorewall firewalls to block that IP Address from accessing or even connecting to your server for a period of time.
Banning these IP Addresses will reduce the amount of traffic to your website however like with the unruly school students in the example above you really don’t want these visitors because their sole purpose is to do harm to your site or server.
Locating your log files.
Most but not all logs in Ubuntu are situated in /var/log/ and the folders under that location. To find all logs on you server got to a terminal window and type
locate log
then hit enter and all logs on your server will be listed.
Open some of these logs and have a close look at their contents. Particularly have a look at /var/log/auth.log, /var/log/mail.log, /var/log/apache2/error.log and the access and error log for each virtual server on your server. When I say open and look at the logs I mean what you want to do is examine them almost forensically. Familiarise your self with the logs get to know the patterns and variations. Looking at these logs can take quite some time at first but be patient, in a short while you will start to see things you need to investigate further,
Explaining how to read your log files.
So lets look at some of the things you need to look for and an expiation of what they mean. The examples below are from my logs. To start with lets look at my apache error log.
Apache error log
[Mon May 23 00:16:47 2011] [error] [client 69.28.58.15] File does not exist: /home/binaryone/public_html/+args[i+1]+, referer: http://www.binaryone.com.au/+args%5bi+1%5d+
[Mon May 23 00:18:06 2011] [error] [client 69.28.58.15] File does not exist: /home/binaryone/public_html/Includes/+args[i+1]+, referer: http://www.binaryone.com.au/Includes/+args%5bi+1%5d+
[Mon May 23 06:26:12 2011] [error] [client 66.249.71.147] File does not exist: /home/binaryone/public_html/WebDesign/Referrals
[Tue May 24 05:53:54 2011] [error] [client 66.249.67.150] File does not exist: /home/binaryone/public_html/WebDesign/WebDesign
[Tue May 24 12:02:39 2011] [error] [client 192.168.0.1] File does not exist: /home/binaryone/public_html/phpmyadmin
[Wed May 25 18:07:01 2011] [error] [client 66.249.72.103] File does not exist: /home/binaryone/public_html/WebDesign/Programming
[Wed May 25 21:31:54 2011] [error] [client 69.28.58.15] File does not exist: /home/binaryone/public_html/+args[i+1]+, referer: http://www.binaryone.com.au/+args%5bi+1%5d+
[Wed May 25 21:33:23 2011] [error] [client 69.28.58.15] File does not exist: /home/binaryone/public_html/Includes/+args[i+1]+, referer: http://www.binaryone.com.au/Includes/+args%5bi+1%5d+
[Fri May 27 23:01:42 2011] [error] [client 205.188.116.13] File does not exist: /home/binaryone/public_html/favicon.ico
[Fri May 27 23:03:58 2011] [error] [client 205.188.116.13] File does not exist: /home/binaryone/public_html/favicon.ico
You will see in the apache error log a lot of these type of errors. Basically what this error means is that the hacker has tried to go to to a page or locate a file that does not exist. Once again though the log tells you the date and time, that there was an error, the IP Address of the hacker and a description of what the error was along with the file that the hacker was trying to access.
Notice how there is a pattern to the log entries. This is a common theme in logs which is what will allow fail2ban to carry out it’s work. Pattern matching is something that I will cover in some detail later but for now what you need to do is look for the patterns in your log files.
Now lets take a look at a few records from my access log.
Apache access log
208.115.111.67 – – [28/May/2011:00:44:05 +0800] “GET /robots.txt HTTP/1.1” 200 358 “-” “Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)”
208.115.111.67 – – [28/May/2011:01:33:53 +0800] “GET /WebDesign/WebDesign.php HTTP/1.1” 200 19388 “-” “Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)”
67.195.113.237 – – [28/May/2011:02:40:44 +0800] “GET /robots.txt HTTP/1.0” 200 372 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”
67.195.113.237 – – [28/May/2011:02:40:45 +0800] “GET /Legal/Privacy.php?ReturnPage=../index.php HTTP/1.0” 200 2749 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”
67.195.113.237 – – [28/May/2011:02:40:46 +0800] “GET /Styles/Styles.css HTTP/1.0” 304 176 “http://www.binaryone.com.au/Legal/Privacy.php?ReturnPage=../index.php” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”
88.114.121.193 – – [28/May/2011:04:08:46 +0800] “GET /robots.txt HTTP/1.0” 200 351 “-” “Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)”
88.114.121.193 – – [28/May/2011:04:08:48 +0800] “GET / HTTP/1.1” 200 1739 “-” “Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)”
109.230.216.221 – – [28/May/2011:04:11:00 +0800] “GET /WebDesign/QuoteForm.php HTTP/1.1” 302 555 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:04:11:01 +0800] “GET /index.php HTTP/1.1” 200 10688 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:04:11:04 +0800] “GET /Legal/Copyright.php?ReturnPage=../index.php HTTP/1.1” 200 12707 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:04:11:06 +0800] “GET /Legal/Disclamer.php?ReturnPage=../index.php HTTP/1.1” 200 15280 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:04:11:33 +0800] “GET /Legal/Privacy.php?ReturnPage=../index.php HTTP/1.1” 200 12610 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:04:11:38 +0800] “GET /Legal/Terms.php?ReturnPage=../index.php HTTP/1.1” 200 17858 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:04:30:48 +0800] “GET /WebDesign/QuoteForm.php HTTP/1.1” 302 555 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:04:30:50 +0800] “GET /index.php HTTP/1.1” 200 10688 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
77.88.26.25 – – [28/May/2011:08:37:16 +0800] “GET /robots.txt HTTP/1.1” 200 389 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”
109.230.216.221 – – [28/May/2011:10:31:48 +0800] “GET /WebDesign/QuoteForm.php HTTP/1.1” 302 555 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:10:31:49 +0800] “GET /index.php HTTP/1.1” 200 10688 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:10:32:01 +0800] “GET /Legal/Copyright.php?ReturnPage=../index.php HTTP/1.1” 200 12707 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:10:32:07 +0800] “GET /Legal/Disclamer.php?ReturnPage=../index.php HTTP/1.1” 200 15280 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:10:32:11 +0800] “GET /Legal/Privacy.php?ReturnPage=../index.php HTTP/1.1” 200 12610 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:10:32:14 +0800] “GET /Legal/Terms.php?ReturnPage=../index.php HTTP/1.1” 200 17858 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:10:51:42 +0800] “GET /WebDesign/QuoteForm.php HTTP/1.1” 302 555 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
109.230.216.221 – – [28/May/2011:10:51:43 +0800] “GET /index.php HTTP/1.1” 200 10688 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
119.63.196.56 – – [28/May/2011:10:55:40 +0800] “GET /robots.txt HTTP/1.1” 200 372 “-” “Baiduspider+(+http://www.baidu.com/search/spider.htm)”
66.249.72.176 – – [28/May/2011:11:35:09 +0800] “GET /robots.txt HTTP/1.1” 200 410 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.72.176 – – [28/May/2011:11:35:10 +0800] “GET / HTTP/1.1” 200 1113 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
67.195.113.237 – – [28/May/2011:13:19:06 +0800] “GET /WebDesign/WebDesign.php HTTP/1.0” 200 3310 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”
67.195.113.237 – – [28/May/2011:13:19:07 +0800] “GET /Styles/Styles.css HTTP/1.0” 304 176 “http://www.binaryone.com.au/WebDesign/WebDesign.php” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”
In the access log all the familiar elements are there again but this time in a slightly different order. Notice how this time the IP Address is the first cab of the rank. Then the date and time followed by what we will call a get statement surrounded by inverted comers and then two numbers. Then after the two numbers is another set of inverted comers with information between them followed by yet another set with different information. Do you see a pattern starting to emerge?
Log entries explained in detail.
The IP Address, date and time are self explanatory so lets start with the get statement. The get statement could actually be a GET, POST, HEAD, INSERT statement but either way what this entry is telling you is that the user is trying to do something with the file named in the statement. What that something is depends on the GET, POST, HEAD and INSERT.
GET means they are looking at the page or post. POST means that they are trying to enter data in your database. HEAD means that they are not requesting the entire page, just the header of the page usually to compare the date to see if the page has been updated since they last looked at the page or post. INSERT means they are trying to write over the page with there own code replacing your page.
The two sets of numbers that follow these statements are the result of what happened. The first number is the apache response code. The most common response code should be 200 which means the file the user requested is available. If you get a 404 response code then the file was not available. If it is a file that you would expect to be there you need to see what the problem is and fix it. But be aware that people or scripts (written by people) will test your server to see if certain pages exist and try to exploit some vulnerability that lay on the page they are testing for. For a list of apache response codes look at my post Where to find Apache response codes and how to determine their meaning. The second number is the amount of data that was transferred back to the user by your server in bites.
That bring us to the two sets of inverted comers with information between them. The first set of inverted comers holds a variety of things including the page that the users was on when the request was made or a “-” if the correct page was returned. But the second set of inverted comers should tell you the browser the user was using and if the user is a spider or a bot then it should contain the name of the bot. But don’t rely on this with out checking because some hackers will give there bots the same name as others in an attempt to mislead the unwary. However there are hackers that don’t care and those who are proud of being able to hack your server. So they will name their bot so you will know their work.
Finally for the logs, take a look at my mail log.
Mail Log
May 1 12:35:51 Myserver postfix/anvil[14462]: statistics: max connection rate 1/60s for (smtp:220.233.2.69) at May 1 12:32:30
May 1 12:35:51 Myserver postfix/anvil[14462]: statistics: max connection count 1 for (smtp:220.233.2.69) at May 1 12:32:30
May 1 12:35:51 Myserver postfix/anvil[14462]: statistics: max cache size 1 at May 1 12:32:30
May 1 12:39:04 Myserver postfix/smtpd[14742]: connect from unknown[190.233.96.177]
May 1 12:39:06 Myserver postfix/smtpd[14742]: NOQUEUE: reject: RCPT from unknown[190.233.96.177]: 550 5.1.1 <allan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<3dadeverson@chcaustralia.com> to=<allan@binaryone.com.au> proto=ESMTP helo=<[190.233.96.177]>
May 1 12:39:06 Myserver postfix/smtpd[14742]: NOQUEUE: reject: RCPT from unknown[190.233.96.177]: 550 5.1.1 <windowsallanallan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<3dadeverson@chcaustralia.com> to=<windowsallanallan@binaryone.com.au> proto=ESMTP helo=<[190.233.96.177]>
May 1 12:39:06 Myserver postfix/smtpd[14742]: NOQUEUE: reject: RCPT from unknown[190.233.96.177]: 550 5.1.1 <memberallan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<3dadeverson@chcaustralia.com> to=<memberallan@binaryone.com.au> proto=ESMTP helo=<[190.233.96.177]>
May 1 12:39:07 Myserver postfix/smtpd[14742]: disconnect from unknown[190.233.96.177]
May 1 12:42:27 Myserver postfix/anvil[14758]: statistics: max connection rate 1/60s for (smtp:190.233.96.177) at May 1 12:39:04
May 1 12:42:27 Myserver postfix/anvil[14758]: statistics: max connection count 1 for (smtp:190.233.96.177) at May 1 12:39:04
May 1 12:42:27 Myserver postfix/anvil[14758]: statistics: max cache size 1 at May 1 12:39:04
May 1 13:13:30 Myserver postfix/smtpd[15691]: warning: 190.51.202.228: hostname 190-51-202-228.speedy.com.ar verification failed: Name or service not known
May 1 13:13:30 Myserver postfix/smtpd[15691]: connect from unknown[190.51.202.228]
May 1 13:13:32 Myserver postfix/smtpd[15694]: connect from unknown[187.1.218.28]
May 1 13:13:35 Myserver postfix/smtpd[15691]: NOQUEUE: reject: RCPT from unknown[190.51.202.228]: 550 5.1.1 <allan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<info@biniflighttraining.com.au> to=<allan@binaryone.com.au> proto=ESMTP helo=
May 1 13:13:36 Myserver postfix/smtpd[15691]: NOQUEUE: reject: RCPT from unknown[190.51.202.228]: 550 5.1.1 <windowsallanallan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<rj@binaryculture.com.au> to=<windowsallanallan@binaryone.com.au> proto=ESMTP helo=
May 1 13:13:37 Myserver postfix/smtpd[15691]: NOQUEUE: reject: RCPT from unknown[190.51.202.228]: 550 5.1.1 <memberallan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<windowsallanallan@binaryone.com.au> to=<memberallan@binaryone.com.au> proto=ESMTP helo=
May 1 13:13:40 Myserver postfix/smtpd[15694]: NOQUEUE: reject: RCPT from unknown[187.1.218.28]: 550 5.1.1 <allan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<taxdwhxkb@ancientmu.com> to=<allan@binaryone.com.au> proto=SMTP helo=
May 1 13:13:42 Myserver postfix/smtpd[15694]: disconnect from unknown[187.1.218.28]
May 1 13:13:50 Myserver postfix/smtpd[15691]: disconnect from unknown[190.51.202.228]
May 1 13:17:10 Myserver postfix/anvil[15693]: statistics: max connection rate 1/60s for (smtp:190.51.202.228) at May 1 13:13:30
May 1 13:17:10 Myserver postfix/anvil[15693]: statistics: max connection count 1 for (smtp:190.51.202.228) at May 1 13:13:30
May 1 13:17:10 Myserver postfix/anvil[15693]: statistics: max cache size 2 at May 1 13:13:32
May 1 13:49:27 Myserver postfix/smtpd[16751]: connect from unknown[200.96.37.194]
May 1 13:49:28 Myserver postfix/smtpd[16751]: NOQUEUE: reject: RCPT from unknown[200.96.37.194]: 550 5.1.1 <allan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<amphitheatrest05@3hgroup.com> to=<allan@binaryone.com.au> proto=ESMTP helo=<[200.96.37.194]>
May 1 13:49:28 Myserver postfix/smtpd[16751]: NOQUEUE: reject: RCPT from unknown[200.96.37.194]: 550 5.1.1 <windowsallanallan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<amphitheatrest05@3hgroup.com> to=<windowsallanallan@binaryone.com.au> proto=ESMTP helo=<[200.96.37.194]>
May 1 13:49:28 Myserver postfix/smtpd[16751]: NOQUEUE: reject: RCPT from unknown[200.96.37.194]: 550 5.1.1 <memberallan@binaryone.com.au>: Recipient address rejected: User unknown in virtual alias table; from=<amphitheatrest05@3hgroup.com> to=<memberallan@binaryone.com.au> proto=ESMTP helo=<[200.96.37.194]>
May 1 13:49:29 Myserver postfix/smtpd[16751]: disconnect from unknown[200.96.37.194]
By this time you should be able to look at this log and see the obvious details that we have covered previously. So I am not going to cover them again. Take a look at the sections of the log that look like this “reject: RCPT from unknown[200.96.37.194]: 550 5.1.1 ”. This is a particularly insidious type of hack attempt. Basically what the hacker is doing, is sending mail with forged sender addresses to your server. Then your server replies to an innocent server with undeliverable mail notifications. The hacker tries to use lots of servers to flood the innocent server, causing a denial of service attack. This is called backscatter mail. For more information on backscatter mail see http://www.postfix.org/BACKSCATTER_README.html
As you should be able to see, hackers will try many ways to cause problems with your server or try to use your server to attack another innocent server. The trick is finding a way to stop them in their tracks. Apart from sitting all day reading your log files and taking the appropriate actions what you need is a program to look after it for you. That is where fail2ban comes in.
Over the coming posts I will look at the configuration for fail2ban to look after the logs for you.