System intrusion detection, primarily on linux servers wit..

Damo Gets · External Since: Jan 14, 2009 Posts: 1

Okay, we've got some serious issues going on here at work. We've lost
the dude that knew everything about how our network was set up; myself
and my cow orker are working full time on recovering all of the
information that we can and now we've got the added task of hardening
our systems from outside attack.

Right now I need to know if any of you *NIX networking ppl and/or
sysadmins can possibly recommend an intrusion detection system/suite.
I've heard that 'snort' is the de facto standard. We're administering
a handful of windows machines, a load of distributed Linux servers
over a WAN with 3 gateways to the outside world + a T1, 2 Linux
asterisk PBX servers, and some laptops that occasionally VPN in. I'd
at the very least like to be able to monitor all of our linux systems
in a centralized manner.

To this point I've been utilizing nagios, mrtg, and ntop. As our
potential cracker realizes how we've identified him in bandwidth
spikes, identifying compromised binaries by modification dates, and
top on any potentially compromised machines to watch for processes
spawned.

I'm adding syslogging of login attempts, etc, but the dude that got
into our WAN is sophisticated. He'll be covering his tracks better in
no time. And honestly, keeping a tail -f /var/log/messages open on
each server and my eyeballs glued to bandwidth graphs updating at 5
minute intervals will not be an option much longer.

Anybody got any ideas and/or need more information to be able to give
some helpful hints?

TIA

C. · External Since: Nov 19, 2006 Posts: 6

On 15 Jan, 04:00, Damo Gets <dgets... DeleteThis @amirehab.net> wrote:
> Okay, we've got some serious issues going on here at work. We've lost
> the dude that knew everything about how our network was set up; myself
> and my cow orker are working full time on recovering all of the
> information that we can and now we've got the added task of hardening
> our systems from outside attack.
>
> Right now I need to know if any of you *NIX networking ppl and/or
> sysadmins can possibly recommend an intrusion detection system/suite.
> I've heard that 'snort' is the de facto standard. We're administering
> a handful of windows machines, a load of distributed Linux servers
> over a WAN with 3 gateways to the outside world + a T1, 2 Linux
> asterisk PBX servers, and some laptops that occasionally VPN in. I'd
> at the very least like to be able to monitor all of our linux systems
> in a centralized manner.
>
> To this point I've been utilizing nagios, mrtg, and ntop. As our
> potential cracker realizes how we've identified him in bandwidth
> spikes, identifying compromised binaries by modification dates, and
> top on any potentially compromised machines to watch for processes
> spawned.
>
> I'm adding syslogging of login attempts, etc, but the dude that got
> into our WAN is sophisticated. He'll be covering his tracks better in
> no time. And honestly, keeping a tail -f /var/log/messages open on
> each server and my eyeballs glued to bandwidth graphs updating at 5
> minute intervals will not be an option much longer.
>
> Anybody got any ideas and/or need more information to be able to give
> some helpful hints?
>
> TIA

I'd stay well clear of network IDS until you've got your host based
IDS sorted out (tripwire, L5, rootkit hunter). The former might tell
you about stuff hitting your network (and a lot of false positives)
but the latter will allow you to detect and recover from a successful
attack. Currently, if you suspect your systems have been compromised
then you can't rely on any executable/library/code - time to reformat
those disks.

> To this point I've been utilizing nagios, mrtg, and ntop. As our
> potential cracker realizes how we've identified him in bandwidth
> spikes, identifying compromised binaries by modification dates, and
> top on any potentially compromised machines to watch for processes
> spawned.

Do you think modification dates can't be tampered with? Processes
hidden? Certainly, its a starting point.

C.

Bit Twister · External Since: Dec 19, 2004 Posts: 1894

On Wed, 14 Jan 2009 20:00:58 -0800 (PST), Damo Gets wrote:

> Right now I need to know if any of you *NIX networking ppl and/or
> sysadmins can possibly recommend an intrusion detection system/suite.
> I've heard that 'snort' is the de facto standard.

http://sourceforge.net/projects/aide
http://la-samhna.de/samhain/s_documentation.html
http://www.ossec.net/

1PW · External Since: Jan 15, 2009 Posts: 6

On 01/14/2009 08:00 PM, Damo Gets sent:

> a load of distributed Linux servers

Exactly which Linux OSs are you supporting on these servers?
Distro, version...

Pete
--
1PW @?6A62?FEH9:DE=6o2@=]4@> [r4o7t]

Moe Trin · External Since: Aug 12, 2004 Posts: 1731

On Wed, 14 Jan 2009, in the Usenet newsgroup comp.os.linux.networking in
article <d64c5d86-7428-456e-a5da-d3ff895afd2e.DeleteThis@v18g2000pro.googlegroups.com>
and in the Usenet newsgroup comp.os.linux.security, in article
<cb3fe989-0064-4875-97c2-ba61d5c3e102.DeleteThis@40g2000prx.googlegroups.com>,
Damon Getsman wrote:

NOTE: Posting from groups.google.com (or some web-forums) dramatically
reduces the chance of your post being seen. Find a real news server.

NOTE: Please don't multipost the same message to multiple groups. If
you MUST post to multiple groups, include them all in the Newsgroups:
header, and set a Followup-To: header as I have done here.

>As our potential cracker realizes how we've identified him in
>bandwidth spikes, identifying compromised binaries by modification
>dates, and top on any potentially compromised machines to watch for
>processes spawned.

Are you indicating that someone already _HAS_ gotten in? If so, you
are screwed - as you don't know what that person has done. You mention
identifying compromised binaries - that implies the person has root -
and think you can identify things by modification date. Boy, are you
ever wrong:

[ibuprofin ~]$ ls FOO
ls: FOO: No such file or directory
[ibuprofin ~]$ touch FOO
[ibuprofin ~]$ ls -l FOO
-rw-r--r-- 1 ibuprofi users 0 Jan 15 12:28 FOO
[ibuprofin ~]$ touch -t 102005152001.13 FOO
[ibuprofin ~]$ ls -l --full-time FOO
-rw-r--r-- 1 ibuprofi users 0 Sat Oct 20 05:15:13 2001 FOO
[ibuprofin ~]$

>I'm adding syslogging of login attempts, etc, but the dude that got
>into our WAN is sophisticated. <A0>He'll be covering his tracks better
>in no time.

And the bad guy is getting in exactly how? You're posting from an
address of a company in the medical business. If there is even the
remote possibility that this is subject to HIPAA, disconnect the network
from the Internet RIGHT NOW.

>And honestly, keeping a tail -f /var/log/messages open on each server
>and my eyeballs glued to bandwidth graphs updating at 5 minute
>intervals will not be an option much longer.

as it's a waste of time if the bad guy has gotten in once already. How
do you know that the tools you are using haven't been 0wn3d already?

You're posting from what is indicating Ubunto 8.04. If that is
representative of your Linux, have you been keeping things up to date?
If so, the bad guy may be getting in through a compromised account.
Which one?

In the Usenet newsgroup comp.os.linux.security, Bit Twister has
suggested aide and samhain (and ossec which I have no experience with).
A problem with these tools is that they are best used starting with a
"clean" system, and allowing them to report on changes. In both Usenet
newsgroups, others have recommended a reinstall. I TOTALLY agree with
that recommendation. If you believe you've been compromised, the _only_
solution is a wipe and reinstall, followed by a full update before
allowing access from the world.

Hmmm, I wonder if your SSH setup is one of the compromised ones. From
the US-CERT Technical Cyber Security Alert TA08-137A from last May:

Overview

A vulnerability in the OpenSSL package included with the Debian
GNU/Linux operating system and its derivatives may cause weak
cryptographic keys to be generated. Any package that uses the affected
version of SSL could be vulnerable.

I. Description

A vulnerabiliity exists in the random number generator used by the
OpenSSL package included with the Debian GNU/Linux, Ubuntu, and other
Debian-based operating systems. This vulnerability causes the
generated numbers to be predictable.

That was patched back in May, but any keys generated before then are
vulnerable. There are exploits of this in the wild.

Old guy

Damon Getsman · External Since: Jan 16, 2009 Posts: 1

> NOTE: Posting from groups.google.com (or some web-forums) dramatically
> reduces the chance of your post being seen. Find a real news server.

Thank you for this information; however, the ISPs that we are using
for the gateway that I'm forced to use at this point prevent me from
having access to a decent raw nntp server AFAIK. Once I'm not swamped
with other things I'll check this out.

> NOTE: Please don't multipost the same message to multiple groups. If
> you MUST post to multiple groups, include them all in the Newsgroups:
> header, and set a Followup-To: header as I have done here.

I normally do. It was a mistake on my part.

> Are you indicating that someone already _HAS_ gotten in? If so, you
> are screwed - as you don't know what that person has done. You mention
> identifying compromised binaries - that implies the person has root -
> and think you can identify things by modification date. Boy, are you
> ever wrong:

I am aware of that, and yes, I'm pretty sure that he did get root.
The modification stamps on /bin/bash and /bin/grep didn't get there by
anyone else and I didn't do it personally. It does tell a little bit
about the sophistication level of the person that I'm dealing with,
though.

I am fully aware of the ability to modify datestamps by 'touch' and
hex filesystem access on a low level. Obviously it either indicates
that the person thought we were incompetent (or did not have enough
resources available as a target) enough to not bother with the time
for that, or they do not posses the level of sophistication to
implement that.

> And the bad guy is getting in exactly how? You're posting from an
> address of a company in the medical business. If there is even the
> remote possibility that this is subject to HIPAA, disconnect the network
> from the Internet RIGHT NOW.

Almost certainly looks to be from an apache or webmin hole. I
attempted to close these things a long time ago and was hamstrung by
someone over my head that has been dealt with for this already. I'm
making sure that I don't make the same mistakes again.

> as it's a waste of time if the bad guy has gotten in once already. How
> do you know that the tools you are using haven't been 0wn3d already?

I know, that's why I'm looking for something better on the next way
around.

> You're posting from what is indicating Ubunto 8.04. If that is
> representative of your Linux, have you been keeping things up to date?
> If so, the bad guy may be getting in through a compromised account.
> Which one?

We haven't been (AGAINST my advice, which is why the aforementioned
people are being 'dealt with'), and if he's got a brain he's snagged
our password and shadow files, so we're considering everything
compromised. I've dealt with network security from the other side of
the spectrum when I was an angsty teenager and learned how
exploitations occur. So I may not have the knowledge to deal with
this on the administration side just yet, but I do know what I'm
looking for, what he's capable of, and have an idea of what we're
playing with.

> In the Usenet newsgroup comp.os.linux.security, Bit Twister has
> suggested aide and samhain (and ossec which I have no experience with).
> A problem with these tools is that they are best used starting with a
> "clean" system, and allowing them to report on changes. In both Usenet
> newsgroups, others have recommended a reinstall. I TOTALLY agree with
> that recommendation. If you believe you've been compromised, the _only_
> solution is a wipe and reinstall, followed by a full update before
> allowing access from the world.

Thank you for the advice. I'll look into the tools. I'm doing what I
can to advocate what you've mentioned, but I might be hamstrung from
higher up in being able to do a full cleaning.

> Hmmm, I wonder if your SSH setup is one of the compromised ones. From
> the US-CERT Technical Cyber Security Alert TA08-137A from last May:

On the old machine it certainly was. I was aware of this ssl
vulnerability but didn't think of it because I didn't know about
Asterisk's usage of SSL. I'll look into it more now. As soon as that
issue was mentioned I made sure to update all of our servers to non-
vulnerable implementation and keyset.

Thank you much, I appreciate the time and consideration that you put
into your message a lot. I don't have a whole lot of resources to
work with right now and a pointing in the right direction saves me a
lot of time and enables me to do what I can over the weekend when all
of our users are off of the main servers, so quick action is a
necessity.

-D

Moe Trin · External Since: Aug 12, 2004 Posts: 1731

On Fri, 16 Jan 2009, in the Usenet newsgroups comp.os.linux.networking and
comp.os.linux.security, in article
<b155cbc2-869a-4137-9e83-5c948391028a.TakeThisOut@v18g2000pro.googlegroups.com>, Damon
Getsman wrote:

NOTE: Posting from groups.google.com (or some web-forums) dramatically
reduces the chance of your post being seen. Find a real news server.

>Once I'm not swamped with other things I'll check this out.

Understood - try http://www.dmoz.org/Computers/Usenet/Public_News_Servers/
when you have time. The 'NOTE:' is added automatically by my news tool
when I'm replying to a googlegroups post.

>> Are you indicating that someone already _HAS_ gotten in? If so, you
>> are screwed - as you don't know what that person has done.

>I am aware of that, and yes, I'm pretty sure that he did get root.

The only safe solution is to wipe and reinstall from known clean media.
But you know that too.

>The modification stamps on /bin/bash and /bin/grep didn't get there by
>anyone else and I didn't do it personally. It does tell a little bit
>about the sophistication level of the person that I'm dealing with,
>though.

For informational use only - look at your package manager. 'rpm' has a
'-Va' option, and Debian based systems have 'debsums'. Both will do a
md5sum check against a baseline (that may or may not be compromised).
Neither is good enough to use as a IDS, and the 'debsums' man page even
warns about that. Something as simple as a MD5 hash isn't solid, given
that the /usr/bin/md5sum may be ``fixed'' and the database itself
modified if it's accessible. See below.

>Obviously it either indicates that the person thought we were
>incompetent (or did not have enough resources available as a target)
>enough to not bother with the time for that, or they do not posses the
>level of sophistication to implement that.

That may be true, but I don't think you can risk it either.

>> And the bad guy is getting in exactly how?

>Almost certainly looks to be from an apache or webmin hole.

Yeah, there are plenty of those. Two points: PHP is generally a walking
disaster area - looking at Bugtraq shows more PHP exploits than
anything else, followed fairly closely by SQL.

Second point, if one of your users has been 0wn3d _elsewhere_ and they
had their SSH authentication tokens on "that" system, there is an
exploit out there that uses stolen authentication to get "in" to your
system, uses any number of local exploits to escalate privilege to
root, and then installs the 'Phalanx2' root kit that hides itself
pretty well, and searches for authentication data on your system (which
it mails out to a drop-box). CERT reported this about five months ago.

>I've dealt with network security from the other side of the spectrum
>when I was an angsty teenager and learned how exploitations occur. So
>I may not have the knowledge to deal with this on the administration
>side just yet, but I do know what I'm looking for, what he's capable
>of, and have an idea of what we're playing with.

The only safe solution is the wipe/reinstall/update. There are a
number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
(http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
to look for many rootkits. By in large, these are trivial to defeat or
bypass. As one example, all look for the existence of a file named
'/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
sniffer worm from 2003). That's all well and good, but what if the
mal-ware author does something terribly complicated, such as changing
the filename to '/tmp/.../A' or '/tmp/.../b' or something... yup, the
anti-mal-ware tools won't find it.

>> In the Usenet newsgroup comp.os.linux.security, Bit Twister has
>> suggested aide and samhain (and ossec which I have no experience
>> with). A problem with these tools is that they are best used
>> starting with a "clean" system, and allowing them to report on
>> changes.

Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
strong point, but I'm much less thrilled by it now.

>Thank you for the advice. I'll look into the tools. I'm doing what I
>can to advocate what you've mentioned, but I might be hamstrung from
>higher up in being able to do a full cleaning.

Many of us suffer through the same interference. Best advice remains
a wipe, reinstall from trusted media, and updates. In this case, I'd
also strongly recommend new passwords for ALL accounts. Then, take a
snapshot of the system using 'aide' (or the fore-runner 'tripwire').
This is what the system looks like "now" - and you can then compare
this snapshot to "later" and look for changes. Big caution: the
aide binary and database go on removable media that is kept in a safe
place when not being used. Also, you have to retake the snapshot each
time you intentionally change something (such as security updates). It
is a pain in the a$$, but less of a pain than what you are going
through now.

>> Hmmm, I wonder if your SSH setup is one of the compromised ones. From
> the US-CERT Technical Cyber Security Alert TA08-137A from last May:

>On the old machine it certainly was. I was aware of this ssl
>vulnerability but didn't think of it because I didn't know about
>Asterisk's usage of SSL. I'll look into it more now.

If you are running a "current" distribution, then a lot of this is
taken care of by the distribution update process. What it doesn't
handle is an authentication token weakness, such as the above.

>I don't have a whole lot of resources to work with right now and a
>pointing in the right direction saves me a lot of time and enables me
>to do what I can over the weekend when all of our users are off of
>the main servers, so quick action is a necessity.

One further recommendation is the firewall. As of two days ago, there
were 2,771,249,848 IPv4 addresses in use in the world. You may want to
look very strongly at restricting access to all services (except
perhaps mail, DNS, and the web server) to as small a number of address
ranges as practical. For example, you mentioned webmin (and are not
likely to reinstall that again, but) you are probably not to likely
to be needing access to that service from (not pointing fingers at
them, but using them as an example) Central/South America. That means
you could probably NOT ALLOW (as opposed to blocking) IPv4 addresses
in the 186.x.x.x/7, 189.x.x.x/8, 190.x.x.x/8 and 200.x.x.x/7 ranges.
The same is true for Asia, Europe, and Africa. Now _blocking_ is
difficult (there are some 93,000 networks in the world, and how do
you know which is which), but _allowing_ access from specific ranges
might prove more practical. Case in point, my home firewall allows
access from a /22 and two /24s "outside" because I can't see any
reason to allow connections from you or anyone else that I haven't
approved in advance, and I really don't expect authorized users to be
connecting from Korea, Kenya, Kuwait or Kazakhstan or a lot of other
places either. If access isn't specifically allowed, then the packet
hits the default rule, and is blocked. Much simpler (and safer) than
trying to exclude every address you believe is "bad".

Old guy

dgetsman · External Since: Jan 21, 2009 Posts: 2

In comp.os.linux.security Moe Trin <ibuprofin.TakeThisOut@painkiller.example.tld> wrote:
> NOTE: Posting from groups.google.com (or some web-forums) dramatically
> reduces the chance of your post being seen. Find a real news server.

Obviously I've gotten to that step now, something I had been neglecting
previously. Thanks.

>>> Are you indicating that someone already _HAS_ gotten in? If so, you
>>> are screwed - as you don't know what that person has done.
>
>>I am aware of that, and yes, I'm pretty sure that he did get root.

At this stage I know that he got root on one of our externally facing
machines. This machine has been wiped and reinstalled but is still not
up to the standards that I would like it to be at. It also had an
internal interface which was not securely firewalled or DMZed. Yes, I
understand the horror of that situation, but it was out of my control.
Thanks to that issue I now have the ability to make suggestions that
will be taken much more seriously regarding our handling of network
security in the future. This does not help the fact that I'm still
learning this as I go, however, at least at this level of
sophistication.

> The only safe solution is to wipe and reinstall from known clean media.
> But you know that too.

As a starting point I have wiped and reinstalled my own workstation; I
am not totally sure that I had it locked down before I reconnected it to
the WAN, however, so I may just end up doing that again. I am also
using a startpoint of my own personal laptop which, at this point, is
disconnected from the WAN and I am fairly certain is not compromised at
the moment. It is running a significantly different kernel from the
other machines, as well.

>>Almost certainly looks to be from an apache or webmin hole.
>
> Yeah, there are plenty of those. Two points: PHP is generally a walking
> disaster area - looking at Bugtraq shows more PHP exploits than
> anything else, followed fairly closely by SQL.

Thank you. As we get downtime where our users are not needing vital
network services I will make sure that we concentrate on these areas and
not providing them where they aren't needed, and maintaining the best I
can find to lock them down where they absolutely are needed.

> Second point, if one of your users has been 0wn3d _elsewhere_ and they
> had their SSH authentication tokens on "that" system, there is an
> exploit out there that uses stolen authentication to get "in" to your
> system, uses any number of local exploits to escalate privilege to
> root, and then installs the 'Phalanx2' root kit that hides itself
> pretty well, and searches for authentication data on your system (which
> it mails out to a drop-box). CERT reported this about five months ago.

Yep. I remember similar .rhost exploits and disasters surrounding them
over a decade ago.

> The only safe solution is the wipe/reinstall/update. There are a
> number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
> (http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
> from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
> to look for many rootkits. By in large, these are trivial to defeat or
> bypass. As one example, all look for the existence of a file named
> '/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
> sniffer worm from 2003). That's all well and good, but what if the
> mal-ware author does something terribly complicated, such as changing
> the filename to '/tmp/.../A' or '/tmp/.../b' or something... yup, the
> anti-mal-ware tools won't find it.

I've actually been able to start going through some OSSEC logs at this
point that are showing things of this sort. Of course, I do not know
enough about the specifics of OSSEC administration just yet to be able
to eliminate all of the false positives, however there are a few entries
in them that are leading me to believe that we have significant
intrusion on some if not all of our primary internal servers within the
WAN. IE:

OSSEC HIDS Notification.
2009 Jan 20 15:29:47

Received From: (agentname) 192.168.1.NOU->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event
(rootcheck)."
Portion of the log(s):

Port '41861'(tcp) hidden. Kernel-level rootkit or trojaned version of
netstat.

--END OF NOTIFICATION

> Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
> strong point, but I'm much less thrilled by it now.

I'm also working on getting these alternate tools installed on a
completely clean system right now to get some decent monitoring in
place.

> Many of us suffer through the same interference. Best advice remains
> a wipe, reinstall from trusted media, and updates. In this case, I'd
> also strongly recommend new passwords for ALL accounts. Then, take a
> snapshot of the system using 'aide' (or the fore-runner 'tripwire').
> This is what the system looks like "now" - and you can then compare
> this snapshot to "later" and look for changes. Big caution: the
> aide binary and database go on removable media that is kept in a safe
> place when not being used. Also, you have to retake the snapshot each
> time you intentionally change something (such as security updates). It
> is a pain in the a$$, but less of a pain than what you are going
> through now.

Gotcha. Your recommendations have given me an excellent starting point
and I thank you much. Any further feedback is greatly appreciated.

-Damon Getsman

1PW · External Since: Jan 15, 2009 Posts: 6

On 01/21/2009 07:11 AM, dgetsman RemoveThis @amirehab.net sent:
> In comp.os.linux.security Moe Trin <ibuprofin RemoveThis @painkiller.example.tld> wrote:
>> NOTE: Posting from groups.google.com (or some web-forums) dramatically
>> reduces the chance of your post being seen. Find a real news server.
>
> Obviously I've gotten to that step now, something I had been neglecting
> previously. Thanks.
>
>>>> Are you indicating that someone already _HAS_ gotten in? If so, you
>>>> are screwed - as you don't know what that person has done.
>>> I am aware of that, and yes, I'm pretty sure that he did get root.
>
> At this stage I know that he got root on one of our externally facing
> machines. This machine has been wiped and reinstalled but is still not
> up to the standards that I would like it to be at. It also had an
> internal interface which was not securely firewalled or DMZed. Yes, I
> understand the horror of that situation, but it was out of my control.
> Thanks to that issue I now have the ability to make suggestions that
> will be taken much more seriously regarding our handling of network
> security in the future. This does not help the fact that I'm still
> learning this as I go, however, at least at this level of
> sophistication.
>
>> The only safe solution is to wipe and reinstall from known clean media.
>> But you know that too.
>
> As a starting point I have wiped and reinstalled my own workstation; I
> am not totally sure that I had it locked down before I reconnected it to
> the WAN, however, so I may just end up doing that again. I am also
> using a startpoint of my own personal laptop which, at this point, is
> disconnected from the WAN and I am fairly certain is not compromised at
> the moment. It is running a significantly different kernel from the
> other machines, as well.
>
>>> Almost certainly looks to be from an apache or webmin hole.
>> Yeah, there are plenty of those. Two points: PHP is generally a walking
>> disaster area - looking at Bugtraq shows more PHP exploits than
>> anything else, followed fairly closely by SQL.
>
> Thank you. As we get downtime where our users are not needing vital
> network services I will make sure that we concentrate on these areas and
> not providing them where they aren't needed, and maintaining the best I
> can find to lock them down where they absolutely are needed.
>
>> Second point, if one of your users has been 0wn3d _elsewhere_ and they
>> had their SSH authentication tokens on "that" system, there is an
>> exploit out there that uses stolen authentication to get "in" to your
>> system, uses any number of local exploits to escalate privilege to
>> root, and then installs the 'Phalanx2' root kit that hides itself
>> pretty well, and searches for authentication data on your system (which
>> it mails out to a drop-box). CERT reported this about five months ago.
>
> Yep. I remember similar .rhost exploits and disasters surrounding them
> over a decade ago.
>
>> The only safe solution is the wipe/reinstall/update. There are a
>> number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
>> (http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
>> from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
>> to look for many rootkits. By in large, these are trivial to defeat or
>> bypass. As one example, all look for the existence of a file named
>> '/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
>> sniffer worm from 2003). That's all well and good, but what if the
>> mal-ware author does something terribly complicated, such as changing
>> the filename to '/tmp/.../A' or '/tmp/.../b' or something... yup, the
>> anti-mal-ware tools won't find it.
>
> I've actually been able to start going through some OSSEC logs at this
> point that are showing things of this sort. Of course, I do not know
> enough about the specifics of OSSEC administration just yet to be able
> to eliminate all of the false positives, however there are a few entries
> in them that are leading me to believe that we have significant
> intrusion on some if not all of our primary internal servers within the
> WAN. IE:
>
> OSSEC HIDS Notification.
> 2009 Jan 20 15:29:47
>
> Received From: (agentname) 192.168.1.NOU->rootcheck
> Rule: 510 fired (level 7) -> "Host-based anomaly detection event
> (rootcheck)."
> Portion of the log(s):
>
> Port '41861'(tcp) hidden. Kernel-level rootkit or trojaned version of
> netstat.
>
>
>
> --END OF NOTIFICATION
>
>
>> Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
>> strong point, but I'm much less thrilled by it now.
>
> I'm also working on getting these alternate tools installed on a
> completely clean system right now to get some decent monitoring in
> place.
>
>> Many of us suffer through the same interference. Best advice remains
>> a wipe, reinstall from trusted media, and updates. In this case, I'd
>> also strongly recommend new passwords for ALL accounts. Then, take a
>> snapshot of the system using 'aide' (or the fore-runner 'tripwire').
>> This is what the system looks like "now" - and you can then compare
>> this snapshot to "later" and look for changes. Big caution: the
>> aide binary and database go on removable media that is kept in a safe
>> place when not being used. Also, you have to retake the snapshot each
>> time you intentionally change something (such as security updates). It
>> is a pain in the a$$, but less of a pain than what you are going
>> through now.
>
> Gotcha. Your recommendations have given me an excellent starting point
> and I thank you much. Any further feedback is greatly appreciated.
>
> -Damon Getsman

Hello Damon:

I know you are still trying to "drain the swamp" there. At some point,
I hope you, and those you work for, are able to consider using a fairly
recent enterprise solution such as RHEL5/CentOS5. The addition of
SELinux can help you harden your systems as few other schemes can.

I know that integrating SELinux with Hardy Heron has been done, but it's
a significant undertaking, whereas SELinux has been part of
RHEL/CentOS/Fedora/Debian for awhile.

Below is a URL that points to many guides to system hardening:

<http://www.nsa.gov/ia/guidance/security_configuration_guides/current_guides.shtml>

In the list, you will find hardening tips for Solaris systems.

I immediately realize that the following does not *directly* apply to
*any* of your systems. However, I believe that some of these steps may
still have value for you:

<hXXp://www.nsa.gov/ia/_files/os/redhat/rhel5-pamphlet-i731.pdf>

<hXXp://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf>

I obfuscated the URLs for your security protection.

I'm hoping that some of this will give you food for thought. Good luck.

Pete
--
1PW @?6A62?FEH9:DE=6o2@=]4@> [r4o7t]