DOS Attack & High load

Piero · External Since: Jun 29, 2007 Posts: 2

Hi everyone,

I've a LAMP webserver, with Apache 1.3 and PHP 4, MySQL 4 and Red Hat
Enterprise 4 Update 5.
Assuming the website is www.example.com.

I receive about 20.000 unique users/day. Normally I have about 100
concurrent users and HTTP requests are like:

10.10.10.10 - - [16/Jun/2007:14:26:55 +0200] "GET / HTTP/1.1" 200
48711 "-" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/
20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:55 +0200] "GET /stylesheet.css HTTP/
1.1" 200 8409 "http://www.example.com/" "Mozilla/5.0 (X11; U; Linux
i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET /style2.css HTTP/
1.1" 200 1026 "http://www.example.com/" "Mozilla/5.0 (X11; U; Linux
i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET /style3.css HTTP/
1.1" 200 513 "http://www.example.com/" "Mozilla/5.0 (X11; U; Linux
i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET /images/logo2.gif
HTTP/1.1" 200 4434 "http://www.example.com/" "Mozilla/5.0 (X11; U;
Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-
edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET /images/prova.gif
HTTP/1.1" 200 1831 "http://www.example.com/" "Mozilla/5.0 (X11; U;
Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-
edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET /images/spacer2.gif
HTTP/1.1" 200 43 "http://www.example.com/" "Mozilla/5.0 (X11; U; Linux
i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:57 +0200] "GET /userimgs/first.jpg
HTTP/1.1" 200 21253 "http://www.example.com/" "Mozilla/5.0 (X11; U;
Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-
edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:57 +0200] "GET /images/second.gif
HTTP/1.1" 200 607 "http://www.example.com/" "Mozilla/5.0 (X11; U;
Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-
edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:57 +0200] "GET /images/third.gif
HTTP/1.1" 200 197 "http://www.example.com/" "Mozilla/5.0 (X11; U;
Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-
edgy)"

The system load is 2.00 average (I know, it's high). The problem is
the following. Sometimes I receive HTTP requests like this:

10.10.10.10 - - [15/Jun/2007:23:14:00 +0200] "GET /page.php?id=1 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=2 HTTP/
1.1" 200 16174 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=3 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=4 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=5 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=6 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=7 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=8 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=9 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php?id=10 HTTP/
1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"

or this:

10.10.10.10 - - [15/Jun/2007:23:14:00 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:00 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16174 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"

that are malicious crawling attempts (first case) or DOS attacks
(second case).
In this cases my server load increase to 30-40 because every request
is a query (or more than one because the PHP script query different
tables) and I receive hundreds and hundreds of them.
How can I detect and prevent this?
I tried to use mod_evasive apache module, but it's based on request
per second, so, for mod_evasive there isn't differences between a
normal request (made up by a page and its resources like images, css,
js, ecc) and a DOS attack (just page request) because the number of
requests per second are the same (in my example the number of requests
are 10).

Thanks to everyone and have a great weekend.

buck · External Since: Mar 31, 2004 Posts: 115

On Fri, 29 Jun 2007 03:02:18 -0700, Piero <piero.bacarella.TakeThisOut@gmail.com>
wrote:

>Hi everyone,
>
>I've a LAMP webserver, with Apache 1.3 and PHP 4, MySQL 4 and Red Hat
>Enterprise 4 Update 5.
>Assuming the website is www.example.com.
>
>I receive about 20.000 unique users/day. Normally I have about 100
>concurrent users and HTTP requests are like:
>
>
>10.10.10.10 - - [16/Jun/2007:14:26:55 +0200] "GET / HTTP/1.1" 200
>48711 "-" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/
>20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"

You might try tuning this:

iptables -A HTTP -m state --state NEW -m recent --update \
--seconds 15 -m limit --limit 1/m --limit-burst 1 \
-j LOG --log-prefix "HTTP "
iptables -A HTTP -m state --state NEW -m recent --update \
--seconds 15 --hitcount 3 -j DROP
iptables -A HTTP -m state --state NEW -m recent --set -j ACCEPT
iptables -A HTTP -j ACCEPT \
# Accept what gets through the above

iptables -A INPUT -i $IFE -p tcp --dport 80 -j HTTP \
# Handle HTTP specially
--
buck

Burkhard Ott · External Since: Jun 11, 2007 Posts: 6

Am Fri, 29 Jun 2007 03:02:18 -0700 schrieb Piero:

> Hi everyone,
>
> I've a LAMP webserver, with Apache 1.3 and PHP 4, MySQL 4 and Red Hat
> Enterprise 4 Update 5.
> Assuming the website is www.example.com.
>
> I receive about 20.000 unique users/day. Normally I have about 100
> concurrent users and HTTP requests are like:
> The system load is 2.00 average (I know, it's high). The problem is
> the following. Sometimes I receive HTTP requests like this:
>
> 10.10.10.10 - - [15/Jun/2007:23:14:00 +0200] "GET /page.php?id=1 HTTP/
> 1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
>
> that are malicious crawling attempts (first case) or DOS attacks
> (second case).
> In this cases my server load increase to 30-40 because every request
> is a query (or more than one because the PHP script query different
> tables) and I receive hundreds and hundreds of them.
> How can I detect and prevent this?
> I tried to use mod_evasive apache module, but it's based on request
> per second, so, for mod_evasive there isn't differences between a
> normal request (made up by a page and its resources like images, css,
> js, ecc) and a DOS attack (just page request) because the number of
> requests per second are the same (in my example the number of requests
> are 10).

Do you receive the request really from 10.10.10.10?
You could do iptables -p tcp -s 10.10.10.10 --dport 80 -j REJECT.

Piero · External Since: Jun 29, 2007 Posts: 2

On 29 Giu, 14:12, Burkhard Ott <postmas....DeleteThis@derith.de> wrote:
> Am Fri, 29 Jun 2007 03:02:18 -0700 schrieb Piero:
>
>
>
> > Hi everyone,
>
> > I've a LAMP webserver, with Apache 1.3 and PHP 4, MySQL 4 and Red Hat
> > Enterprise 4 Update 5.
> > Assuming the website iswww.example.com.
>
> > I receive about 20.000 unique users/day. Normally I have about 100
> > concurrent users and HTTP requests are like:
> > The system load is 2.00 average (I know, it's high). The problem is
> > the following. Sometimes I receive HTTP requests like this:
>
> > 10.10.10.10 - - [15/Jun/2007:23:14:00 +0200] "GET /page.php?id=1 HTTP/
> > 1.1" 200 16176 "http://www.example.com/" "Mozilla/4.0 (compatible;
>
> > that are malicious crawling attempts (first case) or DOS attacks
> > (second case).
> > In this cases my server load increase to 30-40 because every request
> > is a query (or more than one because the PHP script query different
> > tables) and I receive hundreds and hundreds of them.
> > How can I detect and prevent this?
> > I tried to use mod_evasive apache module, but it's based on request
> > per second, so, for mod_evasive there isn't differences between a
> > normal request (made up by a page and its resources like images, css,
> > js, ecc) and a DOS attack (just page request) because the number of
> > requests per second are the same (in my example the number of requests
> > are 10).
>
> Do you receive the request really from 10.10.10.10?
> You could do iptables -p tcp -s 10.10.10.10 --dport 80 -j REJECT.

No, it was just an example Smile

mike · External Since: Jul 06, 2007 Posts: 5

iptables -I INPUT -p tcp --src 10.10.10.10 -j DROP

If the attacker is using different IPs, try installing APF then
installing apfados. It might help. It didn't help me on the last
time we were DOSed but I wrote a script to handle that. You can have a
copy of that script (below) it might help. But this script looks for
very specific info in the access log, you'll need to change it
according to your situation. And apf -d just adds an IP into the
firewalls block list. Good luck.

#!/bin/bash

APACHE_LOG=/usr/local/apache/domlogs/xxxxxxxx.info

while true; do
restart=0;
connection_count=`netstat -an|wc -l`;
if [ $connection_count -gt 600 ]; then
for IP in `tail -100 $APACHE_LOG |grep POST | grep -v
profile.php | grep -v \? | awk {'print $1'} | sort -u`; do
restart=1;
apf -d $IP
done
fi
if [ "X$restart" == "X1" ]; then
echo restart apache;
/etc/init.d/httpd stop
sleep 5;
/etc/init.d/httpd start
fi
sleep 10
# Verify apache is running
ps -ef|grep [h]ttp >/dev/null 2>&1
if [ $? -eq 1 ]; then
/etc/init.d/httpd stop
sleep 10;
/etc/init.d/httpd stop
sleep 10;
/etc/init.d/httpd stop
/etc/init.d/httpd startssl
fi
done

mike.TakeThisOut@surgeontech.com