iptables conntrack: packets not matching a rule occasional..

Marc Schiffbauer · External Since: Nov 16, 2004 Posts: 17

Hi all,

I am using iptables 1.2.11 on a well loaded webserver (constant load
of about 3-5).

After revising the firewall policy I found a strange behavior of iptables
that I do not understand:

Most of the packets to port 80 are being ACCEPTed as expected.

But:
* For some clients packets to tcp/80 are being DROPed completely
* For some clients packets to tcp/80 are being DROPed occasionally

I have the following rules for incoming http traffic:
(In OUTPUT everything is allowed ATM)

Chain INPUT (policy DROP 5 packets, 1057 bytes)
pkts bytes target prot opt in out source destination
2227 433K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
54 3048 RULE_2 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 state NEW
[...]
5 200 In_RULE_18 all -- + * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/sec burst 50

Chain RULE_2 (4 references)
pkts bytes target prot opt in out source destination
51 2868 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/sec burst 5 LOG flags 7 level 7 prefix `RULE 2 -- ACCEPT '
54 3048 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain In_RULE_18 (2 references)
pkts bytes target prot opt in out source destination
5 200 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/sec burst 5 LOG flags 7 level 7 prefix `DROP '
5 200 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

(logging for accepted packets has been added for debugging after seeing the dropped
packets...)

When I access the webserver all seems to work fine but I am worried
about these DROPped packets.

The problem is that I cannot see any difference between accepted and
dropped packets :-/

Examples (a-d.x.x.x being 4 different clients and a.b.c.d is my
server; MAC= value was equal in all lines):

Aug 1 15:31:50 pluto kernel: RULE 2 -- ACCEPT IN=eth0 OUT= MAC= SRC=a.x.x.x DST=a.b.c.d LEN=48 TOS=0x00 PREC=0x00 TTL=47 ID=64148 DF PROTO=TCP SPT=52789 DPT=80 WINDOW=5728 RES=0x00 SYN URGP=0
Aug 1 15:31:51 pluto kernel: DROP IN=eth0 OUT= MAC= SRC=b.x.x.x DST=a.b.c.d LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=55422 DF PROTO=TCP SPT=2433 DPT=80 WINDOW=864 RES=0x00 ACK FIN URGP=0
Aug 1 15:31:52 pluto kernel: DROP IN=eth0 OUT= MAC= SRC=c.x.x.x DST=a.b.c.d LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=38797 DF PROTO=TCP SPT=4404 DPT=80 WINDOW=864 RES=0x00 ACK FIN URGP=0
Aug 1 15:31:52 pluto kernel: DROP IN=eth0 OUT= MAC= SRC=c.x.x.x DST=a.b.c.d LEN=40 TOS=0x00 PREC=0x00 TTL=47 ID=2067 DF PROTO=TCP SPT=52789 DPT=80 WINDOW=5728 RES=0x00 ACK URGP=0
Aug 1 15:31:52 pluto kernel: RULE 2 -- ACCEPT IN=eth0 OUT= MAC= SRC=d.x.x.x DST=a.b.c.d LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=47854 DF PROTO=TCP SPT=12768 DPT=80 WINDOW=57344 RES=0x00 SYN URGP=0

I tried it without connection tracking for port 80: No dropped
packets! So it seems related to conntrack

Has anybody an Idea what could be the cause of this?

TIA
-Marc
PS: The same happens with tcp/21 (ftp)
--
8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134

--
To UNSUBSCRIBE, email to debian-isp-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Héctor González · External Since: Aug 01, 2007 Posts: 1

You might try a rule to match "state INVALID", and see if it catches
them. It might be someone probing your firewall.

Marc Schiffbauer wrote:
> Hi all,
>
> I am using iptables 1.2.11 on a well loaded webserver (constant load
> of about 3-5).
>
> After revising the firewall policy I found a strange behavior of iptables
> that I do not understand:
>
> Most of the packets to port 80 are being ACCEPTed as expected.
>
> But:
> * For some clients packets to tcp/80 are being DROPed completely
> * For some clients packets to tcp/80 are being DROPed occasionally
>
> I have the following rules for incoming http traffic:
> (In OUTPUT everything is allowed ATM)
>
> Chain INPUT (policy DROP 5 packets, 1057 bytes)
> pkts bytes target prot opt in out source destination
> 2227 433K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
> 54 3048 RULE_2 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 state NEW
> [...]
> 5 200 In_RULE_18 all -- + * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/sec burst 50
>
> Chain RULE_2 (4 references)
> pkts bytes target prot opt in out source destination
> 51 2868 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/sec burst 5 LOG flags 7 level 7 prefix `RULE 2 -- ACCEPT '
> 54 3048 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
>
> Chain In_RULE_18 (2 references)
> pkts bytes target prot opt in out source destination
> 5 200 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 2/sec burst 5 LOG flags 7 level 7 prefix `DROP '
> 5 200 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
>
> (logging for accepted packets has been added for debugging after seeing the dropped
> packets...)
>
> When I access the webserver all seems to work fine but I am worried
> about these DROPped packets.
>
> The problem is that I cannot see any difference between accepted and
> dropped packets :-/
>
>
> Examples (a-d.x.x.x being 4 different clients and a.b.c.d is my
> server; MAC= value was equal in all lines):
>
> Aug 1 15:31:50 pluto kernel: RULE 2 -- ACCEPT IN=eth0 OUT= MAC= SRC=a.x.x.x DST=a.b.c.d LEN=48 TOS=0x00 PREC=0x00 TTL=47 ID=64148 DF PROTO=TCP SPT=52789 DPT=80 WINDOW=5728 RES=0x00 SYN URGP=0
> Aug 1 15:31:51 pluto kernel: DROP IN=eth0 OUT= MAC= SRC=b.x.x.x DST=a.b.c.d LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=55422 DF PROTO=TCP SPT=2433 DPT=80 WINDOW=864 RES=0x00 ACK FIN URGP=0
> Aug 1 15:31:52 pluto kernel: DROP IN=eth0 OUT= MAC= SRC=c.x.x.x DST=a.b.c.d LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=38797 DF PROTO=TCP SPT=4404 DPT=80 WINDOW=864 RES=0x00 ACK FIN URGP=0
> Aug 1 15:31:52 pluto kernel: DROP IN=eth0 OUT= MAC= SRC=c.x.x.x DST=a.b.c.d LEN=40 TOS=0x00 PREC=0x00 TTL=47 ID=2067 DF PROTO=TCP SPT=52789 DPT=80 WINDOW=5728 RES=0x00 ACK URGP=0
> Aug 1 15:31:52 pluto kernel: RULE 2 -- ACCEPT IN=eth0 OUT= MAC= SRC=d.x.x.x DST=a.b.c.d LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=47854 DF PROTO=TCP SPT=12768 DPT=80 WINDOW=57344 RES=0x00 SYN URGP=0
>
>
> I tried it without connection tracking for port 80: No dropped
> packets! So it seems related to conntrack
>
> Has anybody an Idea what could be the cause of this?
>
> TIA
> -Marc
> PS: The same happens with tcp/21 (ftp)
>

--
Hector Gonzalez
cacho RemoveThis @genac.org
http://www.genac.org

--
To UNSUBSCRIBE, email to debian-isp-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Marc Schiffbauer · External Since: Nov 16, 2004 Posts: 17

* Héctor González schrieb am 01.08.07 um 16:49 Uhr:
> You might try a rule to match "state INVALID", and see if it catches
> them. It might be someone probing your firewall.

makes sense. The new rule matches those packets indeed.

Seems like I did not pay enough attention to the TCP flags.

Thanks Héctor!

-Marc
--
begin LOVE-LETTER-FOR-YOU.txt.vbs
I am a signature virus. Distribute me until the bitter
end

--
To UNSUBSCRIBE, email to debian-isp-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Håkon_Alstadheim · External Since: Sep 14, 2006 Posts: 4

Marc Schiffbauer wrote:
> * Héctor González schrieb am 01.08.07 um 16:49 Uhr:
>
>> You might try a rule to match "state INVALID", and see if it catches
>> them. It might be someone probing your firewall.
>>
>
> makes sense. The new rule matches those packets indeed.
>
> Seems like I did not pay enough attention to the TCP flags.
>
>
Conntrack has a timeout and a limit to the max number of connections it
can remember. I believe it can be adjusted with some setting in /proc or
somewhere. Check the documentation in /usr/src/linux/Documentation.
Anyway, really slow/long-lived web sessions might get caught as invalid
because of this.

--
To UNSUBSCRIBE, email to debian-isp-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Marc Schiffbauer · External Since: Nov 16, 2004 Posts: 17

* Håkon Alstadheim schrieb am 07.08.07 um 23:21 Uhr:
> Marc Schiffbauer wrote:
> >* Héctor González schrieb am 01.08.07 um 16:49 Uhr:
> >
> >>You might try a rule to match "state INVALID", and see if it catches
> >>them. It might be someone probing your firewall.
> >>
> >
> >makes sense. The new rule matches those packets indeed.
> >
> >Seems like I did not pay enough attention to the TCP flags.
> >
> >
> Conntrack has a timeout and a limit to the max number of connections it
> can remember. I believe it can be adjusted with some setting in /proc or
> somewhere. Check the documentation in /usr/src/linux/Documentation.
> Anyway, really slow/long-lived web sessions might get caught as invalid
> because of this.

Sorry I did not mention that I had a look at these values. I think
the default values are ok for http traffic, right?

host:~# for f in /proc/sys/net/ipv4/netfilter/ip_conntrack_*; do
> echo "$f: $(cat $f)"
> done
/proc/sys/net/ipv4/netfilter/ip_conntrack_buckets: 8192
/proc/sys/net/ipv4/netfilter/ip_conntrack_count: 712
/proc/sys/net/ipv4/netfilter/ip_conntrack_generic_timeout: 600
/proc/sys/net/ipv4/netfilter/ip_conntrack_icmp_timeout: 30
/proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid: 0
/proc/sys/net/ipv4/netfilter/ip_conntrack_max: 65536
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal: 0
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_loose: 3
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_max_retrans: 3
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close: 10
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait: 60
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established: 432000
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_fin_wait: 120
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_last_ack: 30
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_max_retrans: 300
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_recv: 60
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_sent: 120
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait: 120
/proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout: 30
/proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream: 180
host:~#

-Marc
--
8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134

--
To UNSUBSCRIBE, email to debian-isp-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Paa.listas · External Since: Aug 08, 2007 Posts: 1

Hello,

echo 110000 > /proc/sys/net/ipv4/netfilter/ip_conntrack_max

110000: put a number suited for your system.

I think this should solve the problem ... can you tell to us if this works
please?

Sorry for my bad English.
Saludos. Pablo.

# -----Mensaje original-----
# De: Marc Schiffbauer [mailto:marc@schiffbauer.net]
# Enviado el: Miércoles, 08 de Agosto de 2007 07:03 a.m.
# Para: debian-isp DeleteThis @lists.debian.org
# Asunto: Re: iptables conntrack: packets not matching a rule occasionally?
#
# * Håkon Alstadheim schrieb am 07.08.07 um 23:21 Uhr:
# > Marc Schiffbauer wrote:
# > >* Héctor González schrieb am 01.08.07 um 16:49 Uhr:
# > >
# > >>You might try a rule to match "state INVALID", and see if it catches
# > >>them. It might be someone probing your firewall.
# > >>
# > >
# > >makes sense. The new rule matches those packets indeed.
# > >
# > >Seems like I did not pay enough attention to the TCP flags.
# > >
# > >
# > Conntrack has a timeout and a limit to the max number of connections it
# > can remember. I believe it can be adjusted with some setting in /proc or
# > somewhere. Check the documentation in /usr/src/linux/Documentation.
# > Anyway, really slow/long-lived web sessions might get caught as invalid
# > because of this.
#
# Sorry I did not mention that I had a look at these values. I think
# the default values are ok for http traffic, right?
#
# host:~# for f in /proc/sys/net/ipv4/netfilter/ip_conntrack_*; do
# > echo "$f: $(cat $f)"
# > done
# /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets: 8192
# /proc/sys/net/ipv4/netfilter/ip_conntrack_count: 712
# /proc/sys/net/ipv4/netfilter/ip_conntrack_generic_timeout: 600
# /proc/sys/net/ipv4/netfilter/ip_conntrack_icmp_timeout: 30
# /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid: 0
# /proc/sys/net/ipv4/netfilter/ip_conntrack_max: 65536
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal: 0
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_loose: 3
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_max_retrans: 3
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close: 10
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait: 60
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established: 432000
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_fin_wait: 120
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_last_ack: 30
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_max_retrans: 300
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_recv: 60
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_sent: 120
# /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait: 120
# /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout: 30
# /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream: 180
# host:~#
#
# -Marc
# --
# 8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134
#

Marc Schiffbauer · External Since: Nov 16, 2004 Posts: 17

* Paa.listas schrieb am 08.08.07 um 20:55 Uhr:
> Hello,

Hi Pablo,

>
>
> echo 110000 > /proc/sys/net/ipv4/netfilter/ip_conntrack_max
>
> 110000: put a number suited for your system.
>
> I think this should solve the problem ... can you tell to us if this works
> please?
>

Thanks for sour answer, but could please you tell me why this should be a
solution? Why 110000?

> Sorry for my bad English.
> Saludos. Pablo.

nm
Thanks for helping!

-Marc
--
+-O . . . o . . . O . . . o . . . O . . . ___ . . . O . . . o .-+
| Ein Service von Links2Linux.de: / o\ RPMs for SuSE |
| --> PackMan! <-- naeheres unter | __| and others |
| http://packman.links2linux.de/ . . . O \__\ . . . O . . . O . |

--
To UNSUBSCRIBE, email to debian-isp-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org