Hacked, now trying to disinfect

joe t. · External Since: Aug 01, 2007 Posts: 2

Yeah, i know, it can't happen in Linux. But it has been happening to
our work servers for several months.

Due to poor security practices of the past catching up to us, three of
our servers (www,mail, and internal business software) got hacked
into, and now there's some bug installed that monitors and logs ssh
logins. It writes login information to /etc/host.

The www and mail servers are running FC6 and Cent4.4, respectively,
and the other is running Slack 10. Suggesting a different distro for
the Slackware box isn't an option at this point. The software that
runs on it is 20+ years old and barely runs even on that OS. The
others seem to work fine aside from the password logger and any other,
more subtle infections present.

i've been looking around, and can't find any references to "/etc/
host" (most links refer to the valid "/etc/hosts" or "host.conf" or
"host.allow/deny" ... Does anyone have any info on this type of
logger? It's clear enough that whoever is doing this is managing to
catch other credentials beyond just the ssh sessions, and the worst
actual damage we've seen has been creating phishing pages. We're
trying to beef up security, but now it's an even steeper uphill battle
with the enemy already inside.

Any help or info on this type of attack would be appreciated.
-joe t.

Dave Uhring · External Since: Apr 17, 2004 Posts: 633

On Wed, 01 Aug 2007 21:07:45 +0000, joe t. wrote:

> Yeah, i know, it can't happen in Linux. But it has been happening to our
> work servers for several months.

Sure it can. Most, probably all, Linux distros are shipped with *root*
login enabled in sshd. If you expose such a system to the Internet you
are almost certain to get successfully attacked.

> i've been looking around, and can't find any references to "/etc/ host"
> (most links refer to the valid "/etc/hosts" or "host.conf" or
> "host.allow/deny" ... Does anyone have any info on this type of logger?
> It's clear enough that whoever is doing this is managing to catch other
> credentials beyond just the ssh sessions, and the worst actual damage
> we've seen has been creating phishing pages. We're trying to beef up
> security, but now it's an even steeper uphill battle with the enemy
> already inside.

Any of that effort is futile. Backup what good data you have and
reinstall, this time blocking root ssh login.

Unruh · External Since: May 27, 2005 Posts: 2192

"joe t." <thookerov DeleteThis @gmail.com> writes:

>Yeah, i know, it can't happen in Linux. But it has been happening to
>our work servers for several months.

Of course it can happen. The usual way is for your password to get hacked
from one of your users. There are password bots out there whcih try to
attack ssh with a guessing attack.

>Due to poor security practices of the past catching up to us, three of
>our servers (www,mail, and internal business software) got hacked
>into, and now there's some bug installed that monitors and logs ssh
>logins. It writes login information to /etc/host.

Yup. The best thing to do now is to a) backup your data, b) do a complete
reformat and reinstall, and c) Do a scan of all of the backups looking for
suid programs.d( change all passwords.ALL.
The hacker knows them all . And once you have done that only then let the
machines back on the net.

>The www and mail servers are running FC6 and Cent4.4, respectively,
>and the other is running Slack 10. Suggesting a different distro for
>the Slackware box isn't an option at this point. The software that
>runs on it is 20+ years old and barely runs even on that OS. The
>others seem to work fine aside from the password logger and any other,
>more subtle infections present.

>i've been looking around, and can't find any references to "/etc/
>host" (most links refer to the valid "/etc/hosts" or "host.conf" or
>"host.allow/deny" ... Does anyone have any info on this type of
>logger? It's clear enough that whoever is doing this is managing to

No they will just grab filenames that look innocuous. Mine had a
/tmp/banana, /dev/cron, and various other files as suid root shells.
(I got broken into because I used telnet and some of my users were in Korea
and got sniffed)
>catch other credentials beyond just the ssh sessions, and the worst
>actual damage we've seen has been creating phishing pages. We're
>trying to beef up security, but now it's an even steeper uphill battle
>with the enemy already inside.

>Any help or info on this type of attack would be appreciated.
>-joe t.

If you want make a system backup that you can study, but first get things
back on track.

Unruh · External Since: May 27, 2005 Posts: 2192

Dave Uhring <daveuhring RemoveThis @yahoo.com> writes:

>On Wed, 01 Aug 2007 21:07:45 +0000, joe t. wrote:

>> Yeah, i know, it can't happen in Linux. But it has been happening to our
>> work servers for several months.

>Sure it can. Most, probably all, Linux distros are shipped with *root*
>login enabled in sshd. If you expose such a system to the Internet you
>are almost certain to get successfully attacked.

Now that is nonesense. You will get attacked, but with a proper password,
the guessing can go on forever.

>> i've been looking around, and can't find any references to "/etc/ host"
>> (most links refer to the valid "/etc/hosts" or "host.conf" or
>> "host.allow/deny" ... Does anyone have any info on this type of logger?
>> It's clear enough that whoever is doing this is managing to catch other
>> credentials beyond just the ssh sessions, and the worst actual damage
>> we've seen has been creating phishing pages. We're trying to beef up
>> security, but now it's an even steeper uphill battle with the enemy
>> already inside.

>Any of that effort is futile. Backup what good data you have and
>reinstall, this time blocking root ssh login.

CptDondo · External Since: Oct 05, 2005 Posts: 309

Unruh wrote:
> Dave Uhring <daveuhring.DeleteThis@yahoo.com> writes:
>
>> On Wed, 01 Aug 2007 21:07:45 +0000, joe t. wrote:
>
>>> Yeah, i know, it can't happen in Linux. But it has been happening to our
>>> work servers for several months.
>
>> Sure it can. Most, probably all, Linux distros are shipped with *root*
>> login enabled in sshd. If you expose such a system to the Internet you
>> are almost certain to get successfully attacked.
>
> Now that is nonesense. You will get attacked, but with a proper password,
> the guessing can go on forever.

Not forever. I had a "strong password" on a system I installed. The
sysadmin failed to notice an attack that started on a Friday afternoon;
by Sunday the system had been compromised. The attack used a
coordinated approach from compromised machines in Romania and Korea, mostly.

Unfortunately the sysadmin also removed the local firewall on that
machine as they had just installed a new hardware firewall, which did
not include a rate-limiter for ssh connections.

*Any* machine can be compromised, given slack enough security in other
areas, even with a strong password, if your pipe is big enough, your CPU
fast enough, and you don't rate-limit new connections.

Randy Yates · External Since: Sep 07, 2006 Posts: 159

CptDondo <yan.RemoveThis@NsOeSiPnAeMr.com> writes:

> Unruh wrote:
>> Dave Uhring <daveuhring.RemoveThis@yahoo.com> writes:
>>
>>> On Wed, 01 Aug 2007 21:07:45 +0000, joe t. wrote:
>>
>>>> Yeah, i know, it can't happen in Linux. But it has been happening to our
>>>> work servers for several months.
>>
>>> Sure it can. Most, probably all, Linux distros are shipped with
>>> *root* login enabled in sshd. If you expose such a system to the
>>> Internet you are almost certain to get successfully attacked.
>> Now that is nonesense. You will get attacked, but with a proper
>> password,
>> the guessing can go on forever.
>
> Not forever. I had a "strong password" on a system I installed. The
> sysadmin failed to notice an attack that started on a Friday
> afternoon; by Sunday the system had been compromised. The attack used
> a coordinated approach from compromised machines in Romania and Korea,
> mostly.

Either they got lucky or your password wasn't that strong.
Here's how I calculated it.

A strong password should be immunune to dictionary attacks. In such a
case, the number of possibilities in an exhaustive search assuming an
8-character password is (52+10+10)^8 = 7.2.E14 password guesses,
assuming 10 symbols are available in addition to 52 letters and 10 numbers.

Now let's assume the machine had a 100 Mbit/sec connection to the internet,
and let's assume that it takes 10 bytes to query and 10 bytes to respond
to the sshd server with a username/password. That means you can make
100E6 / (20* Cool

= 625000 username/password attempts per second.

Assume the password is guessed in 1/100 of the total possible
attempts. Then it would take

(7.22E14 / 100) [password guesses] * 1 sec / (625000 [password guesses])
= 133 days

to guess.

Have I reasoned something incorrectly? If anything, I think I erred
on the side of the hacker.
--
% Randy Yates % "Midnight, on the water...
%% Fuquay-Varina, NC % I saw... the ocean's daughter."
%%% 919-577-9882 % 'Can't Get It Out Of My Head'
%%%% <yates.RemoveThis@ieee.org> % *El Dorado*, Electric Light Orchestra
http://home.earthlink.net/~yatescr

Unruh · External Since: May 27, 2005 Posts: 2192

CptDondo <yan RemoveThis @NsOeSiPnAeMr.com> writes:

>Unruh wrote:
>> Dave Uhring <daveuhring RemoveThis @yahoo.com> writes:
>>
>>> On Wed, 01 Aug 2007 21:07:45 +0000, joe t. wrote:
>>
>>>> Yeah, i know, it can't happen in Linux. But it has been happening to our
>>>> work servers for several months.
>>
>>> Sure it can. Most, probably all, Linux distros are shipped with *root*
>>> login enabled in sshd. If you expose such a system to the Internet you
>>> are almost certain to get successfully attacked.
>>
>> Now that is nonesense. You will get attacked, but with a proper password,
>> the guessing can go on forever.

>Not forever. I had a "strong password" on a system I installed. The
>sysadmin failed to notice an attack that started on a Friday afternoon;
>by Sunday the system had been compromised. The attack used a
>coordinated approach from compromised machines in Romania and Korea, mostly.

I am sorry, but you can only try about 2 passwords per second. Two days is
4x10^5 trials. That is very small even of a 8 character password ( andall
current systems allow an arbitrary length). even at only 40 character, that
is about 10^13 passwords. A strong one would be a random selection so in 2
days the chances of breaking it is 10^-8. Ie, you should consider entering
the lottery.

>Unfortunately the sysadmin also removed the local firewall on that
>machine as they had just installed a new hardware firewall, which did
>not include a rate-limiter for ssh connections.

>*Any* machine can be compromised, given slack enough security in other
>areas, even with a strong password, if your pipe is big enough, your CPU
>fast enough, and you don't rate-limit new connections.

The ssh daemon/pam daemon is not that fast.
10^8 trials per second means you have a terabit network connection to
Romania and Korea. Pretty good.

Scott Hemphill · External Since: Dec 21, 2004 Posts: 27

CptDondo <yan DeleteThis @NsOeSiPnAeMr.com> writes:

> Unruh wrote:
> > Dave Uhring <daveuhring DeleteThis @yahoo.com> writes:
> >
> >> On Wed, 01 Aug 2007 21:07:45 +0000, joe t. wrote:
> >
> >>> Yeah, i know, it can't happen in Linux. But it has been happening to our
> >>> work servers for several months.
> >
> >> Sure it can. Most, probably all, Linux distros are shipped with
> >> *root* login enabled in sshd. If you expose such a system to the
> >> Internet you are almost certain to get successfully attacked.
> > Now that is nonesense. You will get attacked, but with a proper
> > password,
> > the guessing can go on forever.
>
> Not forever. I had a "strong password" on a system I installed. The
> sysadmin failed to notice an attack that started on a Friday
> afternoon; by Sunday the system had been compromised. The attack used
> a coordinated approach from compromised machines in Romania and Korea,
> mostly.
>
> Unfortunately the sysadmin also removed the local firewall on that
> machine as they had just installed a new hardware firewall, which did
> not include a rate-limiter for ssh connections.
>
> *Any* machine can be compromised, given slack enough security in other
> areas, even with a strong password, if your pipe is big enough, your
> CPU fast enough, and you don't rate-limit new connections.

If you can, don't use a password at all. Configure the ssh server to
allow only public key authentication.

Scott
--
Scott Hemphill hemphill DeleteThis @alumni.caltech.edu
"This isn't flying. This is falling, with style." -- Buzz Lightyear

joe t. · External Since: Aug 01, 2007 Posts: 2

On Aug 2, 6:36 am, Axel Werner <axel.wer....DeleteThis@akadpol.bwl.de> wrote:
> joe t. schrieb:
>
> > Yeah, i know, it can't happen in Linux. But it has been happening to
> > our work servers for several months.
>
> > Any help or info on this type of attack would be appreciated.
> > -joe t.
>
> do not disinfect. save/rescue any important data and configurations (no
> binaries!!!!! ASCII configs only, no scripts neither!) and better
> reinstall the whole system. the chance to get back a clean system from a
> hacked one is small and sometimes nearly impossible.
>
> it usualy is easier , safer and faster to reinstall the whole system
> with TRUSTED Installation-Media, Sources and with higher security policies.
>
> then also install and maintenance a host based IDS or some programm that
> tracks changes to important system areas and files in there.. like
> tripwire and similar.
>
> thats just my recommentations.

That's what i thought would end up being the case. i appreciate
everyone's responses. Looks like a long weekend ahead.
-joe t.

Axel Werner · External Since: Jul 11, 2007 Posts: 16

joe t. schrieb:
> Yeah, i know, it can't happen in Linux. But it has been happening to
> our work servers for several months.

>
> Any help or info on this type of attack would be appreciated.
> -joe t.
>

do not disinfect. save/rescue any important data and configurations (no
binaries!!!!! ASCII configs only, no scripts neither!) and better
reinstall the whole system. the chance to get back a clean system from a
hacked one is small and sometimes nearly impossible.

it usualy is easier , safer and faster to reinstall the whole system
with TRUSTED Installation-Media, Sources and with higher security policies.

then also install and maintenance a host based IDS or some programm that
tracks changes to important system areas and files in there.. like
tripwire and similar.

thats just my recommentations.

Captain Dondo · External Since: May 25, 2005 Posts: 300

V Wed, 01 Aug 2007 23:39:58 -0400, Randy Yates napsal(a):

> CptDondo <yan DeleteThis @NsOeSiPnAeMr.com> writes:
>
>> Unruh wrote:
>>> Dave Uhring <daveuhring DeleteThis @yahoo.com> writes:
>>>
>>>> On Wed, 01 Aug 2007 21:07:45 +0000, joe t. wrote:
>>>
>>>>> Yeah, i know, it can't happen in Linux. But it has been happening to our
>>>>> work servers for several months.
>>>
>>>> Sure it can. Most, probably all, Linux distros are shipped with
>>>> *root* login enabled in sshd. If you expose such a system to the
>>>> Internet you are almost certain to get successfully attacked.
>>> Now that is nonesense. You will get attacked, but with a proper
>>> password,
>>> the guessing can go on forever.
>>
>> Not forever. I had a "strong password" on a system I installed. The
>> sysadmin failed to notice an attack that started on a Friday
>> afternoon; by Sunday the system had been compromised. The attack used
>> a coordinated approach from compromised machines in Romania and Korea,
>> mostly.
>
> Either they got lucky or your password wasn't that strong.
> Here's how I calculated it.
>
> A strong password should be immunune to dictionary attacks. In such a
> case, the number of possibilities in an exhaustive search assuming an
> 8-character password is (52+10+10)^8 = 7.2.E14 password guesses,
> assuming 10 symbols are available in addition to 52 letters and 10 numbers.
>
> Now let's assume the machine had a 100 Mbit/sec connection to the internet,
> and let's assume that it takes 10 bytes to query and 10 bytes to respond
> to the sshd server with a username/password. That means you can make
> 100E6 / (20* Cool

= 625000 username/password attempts per second.
>
> Assume the password is guessed in 1/100 of the total possible
> attempts. Then it would take
>
> (7.22E14 / 100) [password guesses] * 1 sec / (625000 [password guesses])
> = 133 days
>
> to guess.
>
> Have I reasoned something incorrectly? If anything, I think I erred
> on the side of the hacker.

Well,they got lucky. The password was *not* a dictionary password, and
was composed of upper and lower case letters. Not entirely random, but
still pretty strong.

My point is, don't bet security on luck.....

That's the only time a system I've worked on got hacked. Multiple layers,
multiple defenses - but I learned and now disable root logins by default
on any exposed system. What I really would like to see is a two-password
option for root, with a timeout for entering the second password and a
timed lockout if multiple attemps fail.

I actually tested that password with JtR and it came up as pretty good....

--Yan

Randy Yates · External Since: Sep 07, 2006 Posts: 159

Captain Dondo <yan DeleteThis @NsOeSiPnAeMr.com> writes:
> [...]
> My point is, don't bet security on luck.....

It's all a matter of luck - the question is, to what degree?

Hey, thanks for sharing your incident. ANY source of information
I get on linux hacks is worth gold to me. Thanks again.
--
% Randy Yates % "Midnight, on the water...
%% Fuquay-Varina, NC % I saw... the ocean's daughter."
%%% 919-577-9882 % 'Can't Get It Out Of My Head'
%%%% <yates DeleteThis @ieee.org> % *El Dorado*, Electric Light Orchestra
http://home.earthlink.net/~yatescr

Dave Uhring · External Since: Apr 17, 2004 Posts: 633

On Thu, 02 Aug 2007 12:25:20 -0400, Randy Yates wrote:

> Even if you closed up root ssh access, I don't see what would stop
> someone from gaining user-level ssh access, and once they had user-level
> ssh access, they could download a root password guesser that would run
> MUCH more quickly on the machine itself (as the user). No? What's to
> keep this from happening?

The difficulty of guessing a user's password is the same as guessing
root's and with a lesser reward for success. In addition the attacker
must also guess a legitimate username whereas the name root is always
there - well, almost always.

Randy Yates · External Since: Sep 07, 2006 Posts: 159

Dave Uhring <daveuhring DeleteThis @yahoo.com> writes:

> On Thu, 02 Aug 2007 12:25:20 -0400, Randy Yates wrote:
>
>> Even if you closed up root ssh access, I don't see what would stop
>> someone from gaining user-level ssh access, and once they had user-level
>> ssh access, they could download a root password guesser that would run
>> MUCH more quickly on the machine itself (as the user). No? What's to
>> keep this from happening?
>
> The difficulty of guessing a user's password is the same as guessing
> root's and with a lesser reward for success. In addition the attacker
> must also guess a legitimate username whereas the name root is always
> there - well, almost always.

Right. Those things are fairly obvious, but once he gains user access,
then isn't it MUCH easier to get access to root?

Also, the OP's lesson leads me to believe that assigning ssh to a
different port number is not worth too much security-wise. If the
access rate to the machine is fast enough, all 65536 ports could be
scanned first for a hot ssh connection in a matter of minutes or even
seconds, no?
--
% Randy Yates % "With time with what you've learned,
%% Fuquay-Varina, NC % they'll kiss the ground you walk
%%% 919-577-9882 % upon."
%%%% <yates DeleteThis @ieee.org> % '21st Century Man', *Time*, ELO
http://home.earthlink.net/~yatescr

Randy Yates · External Since: Sep 07, 2006 Posts: 159

Randy Yates <yates.RemoveThis@ieee.org> writes:

> Dave Uhring <daveuhring.RemoveThis@yahoo.com> writes:
>
>> On Thu, 02 Aug 2007 12:25:20 -0400, Randy Yates wrote:
>>
>>> Even if you closed up root ssh access, I don't see what would stop
>>> someone from gaining user-level ssh access, and once they had user-level
>>> ssh access, they could download a root password guesser that would run
>>> MUCH more quickly on the machine itself (as the user). No? What's to
>>> keep this from happening?
>>
>> The difficulty of guessing a user's password is the same as guessing
>> root's and with a lesser reward for success. In addition the attacker
>> must also guess a legitimate username whereas the name root is always
>> there - well, almost always.
>
> Right. Those things are fairly obvious, but once he gains user access,
> then isn't it MUCH easier to get access to root?

Also, it's much more likely that the username will susceptible to
a dictionary attack.
--
% Randy Yates % "Bird, on the wing,
%% Fuquay-Varina, NC % goes floating by
%%% 919-577-9882 % but there's a teardrop in his eye..."
%%%% <yates.RemoveThis@ieee.org> % 'One Summer Dream', *Face The Music*, ELO
http://home.earthlink.net/~yatescr

Unruh · External Since: May 27, 2005 Posts: 2192

Captain Dondo <yan.RemoveThis@NsOeSiPnAeMr.com> writes:

>Well,they got lucky. The password was *not* a dictionary password, and
>was composed of upper and lower case letters. Not entirely random, but
>still pretty strong.

And I strongly suspect it was NOT brute forcing the root password that got
them in. Probably some sniffed password of a user got them in, and then
they went up to root from there.

>My point is, don't bet security on luck.....

I would bet secutity on a 1/100000000 chance anyday.

>That's the only time a system I've worked on got hacked. Multiple layers,
>multiple defenses - but I learned and now disable root logins by default
>on any exposed system. What I really would like to see is a two-password
>option for root, with a timeout for entering the second password and a
>timed lockout if multiple attemps fail.

VEry bad idea. That opens you up to a DOS attack so suddenly you can no
longer log on as root at all. This they do after they have gotten in, so it
is impossible for you to rescue the system.

>I actually tested that password with JtR and it came up as pretty good....

??? As I said, I suspect is was NOT password brute forcing.

>--Yan

Dave Uhring · External Since: Apr 17, 2004 Posts: 633

On Thu, 02 Aug 2007 14:06:59 -0400, Randy Yates wrote:

> Also, it's much more likely that the username will susceptible to a
> dictionary attack.

There is *no* dictionary attack required for the root account and the
ease of cracking root's password is the same as for a user. Now how
difficult is that to understand?

Have you ever examined the logs of some of those ssh attacks?

Randy Yates · External Since: Sep 07, 2006 Posts: 159

Dave Uhring <daveuhring.TakeThisOut@yahoo.com> writes:

> On Thu, 02 Aug 2007 14:06:59 -0400, Randy Yates wrote:
>
>> Also, it's much more likely that the username will susceptible to a
>> dictionary attack.
>
> There is *no* dictionary attack required for the root account and the
> ease of cracking root's password is the same as for a user. Now how
> difficult is that to understand?

What you say is absolutely true, just like it is true that the
distance from JFK to my sister-in-law's house in Bangalore,
India, is greater than the distance from JFK to the Bangalore
airport.

However, the DIFFERENCE between the two is relatively small.
--
% Randy Yates % "So now it's getting late,
%% Fuquay-Varina, NC % and those who hesitate
%%% 919-577-9882 % got no one..."
%%%% <yates.TakeThisOut@ieee.org> % 'Waterfall', *Face The Music*, ELO
http://home.earthlink.net/~yatescr

Unruh · External Since: May 27, 2005 Posts: 2192

Dave Uhring <daveuhring.TakeThisOut@yahoo.com> writes:

>On Thu, 02 Aug 2007 14:06:59 -0400, Randy Yates wrote:

>> Also, it's much more likely that the username will susceptible to a
>> dictionary attack.

>There is *no* dictionary attack required for the root account and the
>ease of cracking root's password is the same as for a user. Now how
>difficult is that to understand?

Eitehr you are very confused or you express yourself badly.
Dictionary attacks are always the first choice of the attacker. People tend
to choose words as passwords, and any cracker would be an idiot to launch a
full scale brute force without doing dictionary first.
And users are more likely to use easy words simply because they are not as
security concious.

>Have you ever examined the logs of some of those ssh attacks?

Dave Uhring · External Since: Apr 17, 2004 Posts: 633

On Thu, 02 Aug 2007 21:02:47 +0000, Unruh wrote:

> Dave Uhring <daveuhring.RemoveThis@yahoo.com> writes:
>
>>On Thu, 02 Aug 2007 14:06:59 -0400, Randy Yates wrote:
>
>>> Also, it's much more likely that the username will susceptible to a
>>> dictionary attack.
>
>>There is *no* dictionary attack required for the root account and the
>>ease of cracking root's password is the same as for a user. Now how
>>difficult is that to understand?
>
> Eitehr you are very confused or you express yourself badly. Dictionary
> attacks are always the first choice of the attacker.

I was discussing the attack against user account *names* there. No
attack can succeed even against a user account unless the attacker can
first guess a valid account name. The root name requires no guessing.

In *every* attack recorded in my log the root account was attacked first.
And no, I cannot post the log since it was rotated out long ago. In
addition to disabling root login I also have enabled tcpwrappers on sshd
to limit, indeed put an end to, the ssh attacks.

> People tend to
> choose words as passwords, and any cracker would be an idiot to launch a
> full scale brute force without doing dictionary first. And users are
> more likely to use easy words simply because they are not as security
> concious.

Again, the usernames must be guessed, at least those which have shells.
After failure to attack root the typical attack goes after usual system
accounts, which gets them nowhere without a shell. Next they go after
common names in what one might describe as a dictionary attack against
user account names.

Your order of attack is bass ackwards from reality.

>>Have you ever examined the logs of some of those ssh attacks?

The question applies also to you, Bill.