The 'rsh' command always takes 30 seconds

Ramon F Herrera · External Since: Apr 19, 2007 Posts: 30

Hello,

I am writing a client-server system. As customary, the clients side is
on Windows XP and the server on Linux. Since the server program run
fast, and the protocol used is the well-known Berkeley remote shell,
the following is a very good approximation:

C:\> rsh hostname date

Much to my dismay, the above command takes -consistently- 30 seconds
to execute. This sounds like some sort of "tar pit" behind the scenes.

From Linux to Linux, the command takes 1 second.

How can I investigate exactly what is going on?

Any suggestion to accelerate my app is greatly appreciated.

TIA,

-Ramon

Ramon F Herrera · External Since: Apr 19, 2007 Posts: 30

On Jul 14, 1:53 pm, Ramon F Herrera <ra... DeleteThis @conexus.net> wrote:
> Hello,
>
> I am writing a client-server system. As customary, the clients side is
> on Windows XP and the server on Linux. Since the server program run
> fast, and the protocol used is the well-known Berkeley remote shell,
> the following is a very good approximation:
>
> C:\> rsh hostname date
>
> Much to my dismay, the above command takes -consistently- 30 seconds
> to execute. This sounds like some sort of "tar pit" behind the scenes.
>
> From Linux to Linux, the command takes 1 second.
>
> How can I investigate exactly what is going on?
>
> Any suggestion to accelerate my app is greatly appreciated.
>
> TIA,
>
> -Ramon

I forgot to add this: YES, I am fully aware of the rsh issues. This is
a quick prototype, and both the client and the server are being
modified to use ssh. At the time the client was written, ssh was not
available. Currently, the Windows rsh vendor does not support rsh
anymore, which is good. Both the client and the server are in my SOHO.

Thanks for the recommendation, it is always good to remind people to
use secure protocols.

-Ramon

Allen Kistler · External Since: Jun 26, 2004 Posts: 367

Ramon F Herrera wrote:
> I am writing a client-server system. As customary, the clients side is
> on Windows XP and the server on Linux. Since the server program run
> fast, and the protocol used is the well-known Berkeley remote shell,
> the following is a very good approximation:
>
> C:\> rsh hostname date
>
> Much to my dismay, the above command takes -consistently- 30 seconds
> to execute. This sounds like some sort of "tar pit" behind the scenes.

If it was a tar pit, the connection would take infinity seconds to
complete, not 30. It's more likely to be DNS or ident.

> From Linux to Linux, the command takes 1 second.
>
> How can I investigate exactly what is going on?

Look in the server logs; run tcpdump on the server; etc.

> Any suggestion to accelerate my app is greatly appreciated.

Joe Pfeiffer · External Since: Dec 21, 2004 Posts: 94

Ramon F Herrera <ramon.DeleteThis@conexus.net> writes:
>
> I am writing a client-server system. As customary, the clients side is
> on Windows XP and the server on Linux. Since the server program run
> fast, and the protocol used is the well-known Berkeley remote shell,
> the following is a very good approximation:
>
> C:\> rsh hostname date
>
> Much to my dismay, the above command takes -consistently- 30 seconds
> to execute. This sounds like some sort of "tar pit" behind the scenes.
>
> From Linux to Linux, the command takes 1 second.
>
> How can I investigate exactly what is going on?
>
> Any suggestion to accelerate my app is greatly appreciated.

I'd probably start by watching the traffic with wireshark (the earlier
suggestions to use strace, and to not use rsh, are also both good).

Chris Davies · External Since: Apr 13, 2004 Posts: 284

Ramon F Herrera <ramon.TakeThisOut@conexus.net> wrote:
> I am writing a client-server system. As customary, the clients side is
> on Windows XP and the server on Linux. Since the server program run
> fast, and the protocol used is the well-known Berkeley remote shell,

Using rsh is probably a /really/ bad idea. I cannot recommend strongly
enough that you revisit your architectural security analysis (i.e. start
over). Rsh is insecure. Very insecure. Mind-blowingly insecure.

> the following is a very good approximation:
> C:\> rsh hostname date

> the above command takes -consistently- 30 seconds to execute. [...]
> From Linux to Linux, the command takes 1 second.
> How can I investigate exactly what is going on?

Well, if you want to investigate, I suppose you could run strace on the
rsh server and see whether you can determine which calls are blocking.

> Any suggestion to accelerate my app is greatly appreciated.

This sounds very much to me like a typical reverse IP address lookup
is failing. Put more simply, the IP address of your Windows XP client
cannot be resolved to a name by your Linux server, whereas the IP address
of your Linux client can be.

Chris

Unruh · External Since: May 27, 2005 Posts: 2213

Ramon F Herrera <ramon.TakeThisOut@conexus.net> writes:

>Hello,

>I am writing a client-server system. As customary, the clients side is
>on Windows XP and the server on Linux. Since the server program run
>fast, and the protocol used is the well-known Berkeley remote shell,
>the following is a very good approximation:

>C:\> rsh hostname date

>Much to my dismay, the above command takes -consistently- 30 seconds
>to execute. This sounds like some sort of "tar pit" behind the scenes.

rsh is a bad protocol to use these days. It is very insecure. But it
sounds to me like dns problems-- ie your system is trying to lookup
hostname in the dns and is timing out on its first nameserver.
Try doing
rsh 111.222.333.444 date
where the number is the ip address of hostname, and see if that goes
faster. If it does, you know you have dns problems. If not, it is
something else (look in the /var/log/messages or /var/log/syslog for
hints as to when rsh was started.)

>From Linux to Linux, the command takes 1 second.

>How can I investigate exactly what is going on?

>Any suggestion to accelerate my app is greatly appreciated.

Ramon F Herrera · External Since: Apr 19, 2007 Posts: 30

On Jul 14, 3:27 pm, Allen Kistler <ackist... DeleteThis @oohay.moc> wrote:
> Ramon F Herrera wrote:
> > I am writing a client-server system. As customary, the clients side is
> > on Windows XP and the server on Linux. Since the server program run
> > fast, and the protocol used is the well-known Berkeley remote shell,
> > the following is a very good approximation:
>
> > C:\> rsh hostname date
>
> > Much to my dismay, the above command takes -consistently- 30 seconds
> > to execute. This sounds like some sort of "tar pit" behind the scenes.
>

> If it was a tar pit, the connection would take infinity
> seconds to complete, not 30. It's more likely to be
> DNS or ident.

The packet capture contains several 'ident' lines. First time I hear
about it.

-Ramon

Ramon F Herrera · External Since: Apr 19, 2007 Posts: 30

On Jul 14, 1:53 pm, Ramon F Herrera <ra... DeleteThis @conexus.net> wrote:
> Hello,
>
> I am writing a client-server system. As customary, the clients side is
> on Windows XP and the server on Linux. Since the server program run
> fast, and the protocol used is the well-known Berkeley remote shell,
> the following is a very good approximation:
>
> C:\> rsh hostname date
>
> Much to my dismay, the above command takes -consistently- 30 seconds
> to execute. This sounds like some sort of "tar pit" behind the scenes.
>
> From Linux to Linux, the command takes 1 second.
>
> How can I investigate exactly what is going on?
>
> Any suggestion to accelerate my app is greatly appreciated.
>
> TIA,
>
> -Ramon

Hello again,

Apparently, in order for my session to proceed faster, I need one of
these two solutions, both implemented on the Windows (client) side:

(1) I run an 'ident' server on port 113

(2) I modify/configure the Windows TCP/IP stack to return RST (Reset)
as soon as it receives the ident request from the server.

Any suggestions on how to achieve any of the two solutions?

TIA,

-Ramon

Allen Kistler · External Since: Jun 26, 2004 Posts: 367

Ramon F Herrera wrote:
> On Jul 14, 3:27 pm, Allen Kistler <ackist... DeleteThis @oohay.moc> wrote:
>> Ramon F Herrera wrote:
>>> I am writing a client-server system. As customary, the clients side is
>>> on Windows XP and the server on Linux. Since the server program run
>>> fast, and the protocol used is the well-known Berkeley remote shell,
>>> the following is a very good approximation:
>>> C:\> rsh hostname date
>>> Much to my dismay, the above command takes -consistently- 30 seconds
>>> to execute. This sounds like some sort of "tar pit" behind the scenes.
>
> > If it was a tar pit, the connection would take infinity
> > seconds to complete, not 30. It's more likely to be
> > DNS or ident.
>
> The packet capture contains several 'ident' lines. First time I hear
> about it.

So somewhere in the rsh config there's a line to log the remote user ID.
rsh tries for 30 sec to get the user ID out of Windows, but never
does, so then it continues. Your Linux client is either running identd
or sending back error packets, OTOH, so there's no delay. I don't use
rsh, so you'll need to track the config down without my advice.

Personally I disable identd, but some servers require it (usually IRC).
Most servers just desire it and move on when it's not provided.

Ramon F Herrera · External Since: Apr 19, 2007 Posts: 30

On Jul 15, 5:09 am, Dee Earley <dee.ear... RemoveThis @icode.co.uk> wrote:
> On 15/07/2009 05:01, Ramon F Herrera wrote:
>
> > Apparently, in order for my session to proceed faster, I need one of
> > these two solutions, both implemented on the Windows (client) side:
>
> > (1) I run an 'ident' server on port 113
>
> > (2) I modify/configure the Windows TCP/IP stack to return RST (Reset)
> > as soon as it receives the ident request from the server.
>
> > Any suggestions on how to achieve any of the two solutions?
>
> IIRC, windows rejects connections by default (rather than dropping them)
> I'm not 100% sure though.

> If not, you could add an exception for the ident port which
> should then cause the firewall to allow it, and then be rejected
> as nothing is listening.

That is exactly what is happening, Dee. The connection is reject -
after 30 seconds and 4 packets- because nothing is listening. See
here:

http://patriot.net/~ramon/misc/rsh-packet-capture.png

I am trying to prevent the 30 second delay.

There is no firewall, BTW, it is a VPN tunnel which nothing blocked.

-Ramon

Dee Earley · External Since: Jul 15, 2009 Posts: 2

On 15/07/2009 05:01, Ramon F Herrera wrote:
> Apparently, in order for my session to proceed faster, I need one of
> these two solutions, both implemented on the Windows (client) side:
>
> (1) I run an 'ident' server on port 113
>
> (2) I modify/configure the Windows TCP/IP stack to return RST (Reset)
> as soon as it receives the ident request from the server.
>
> Any suggestions on how to achieve any of the two solutions?

IIRC, windows rejects connections by default (rather than dropping them)
I'm not 100% sure though.
If not, you could add an exception for the ident port which should then
cause the firewall to allow it, and then be rejected as nothing is
listening.

--
Dee Earley (dee.earley@icode.co.uk)
i-Catcher Development Team

iCode Systems

Ramon F Herrera · External Since: Apr 19, 2007 Posts: 30

On Jul 15, 9:47 am, Dee Earley <dee.ear....RemoveThis@icode.co.uk> wrote:
> On 15/07/2009 13:30, Ramon F Herrera wrote:
>
>
>
> > On Jul 15, 5:09 am, Dee Earley<dee.ear....RemoveThis@icode.co.uk> wrote:
> >> On 15/07/2009 05:01, Ramon F Herrera wrote:
>
> >>> Apparently, in order for my session to proceed faster, I need one of
> >>> these two solutions, both implemented on the Windows (client) side:
> >>> (1) I run an 'ident' server on port 113
> >>> (2) I modify/configure the Windows TCP/IP stack to return RST (Reset)
> >>> as soon as it receives the ident request from the server.
> >>> Any suggestions on how to achieve any of the two solutions?
> >> IIRC, windows rejects connections by default (rather than dropping them)
> >> I'm not 100% sure though.
>
> > > If not, you could add an exception for the ident port which
> > > should then cause the firewall to allow it, and then be rejected
> > > as nothing is listening.
>
> > That is exactly what is happening, Dee. The connection is reject -
> > after 30 seconds and 4 packets- because nothing is listening. See
> > here:
>
> >http://patriot.net/~ramon/misc/rsh-packet-capture.png
>
> > I am trying to prevent the 30 second delay.
>
> > There is no firewall, BTW, it is a VPN tunnel which nothing blocked.
>
> No it's not, the incoming connections are being dropped, meaning there
> is a firewall on that machine (or you have a broken ident server running)
> As they are being dropped, there is NO reply to the rsh server so the
> only option it has is to wait for a timeout.
>
> To get around this, it MUST be rejected (with a connect failure) by the
> firewall or accepted by an application and dealt with.
> you should be able to add an exception to the windows firewall for that
> port, meaning it then gets rejected rather then silently dropped.
>
> --
> Dee Earley (dee.ear...@icode.co.uk)
> i-Catcher Development Team
>
> iCode Systems

You are absolutely right, Dee. I added the TCP port 113 to the Windows
FW and now things are working.

Thanks!

-Ramon

Chris Davies · External Since: Apr 13, 2004 Posts: 284

Dee Earley <dee.earley.TakeThisOut@icode.co.uk> wrote:
> IIRC, windows rejects connections by default (rather than dropping them)
> I'm not 100% sure though.

The TCP/IP stack will reject them. The default firewall drops them.

> If not, you could add an exception for the ident port which should then
> cause the firewall to allow it, and then be rejected as nothing is
> listening.

This could work, yes.
Chris

Dee Earley · External Since: Jul 15, 2009 Posts: 2

On 15/07/2009 13:30, Ramon F Herrera wrote:
> On Jul 15, 5:09 am, Dee Earley<dee.ear... DeleteThis @icode.co.uk> wrote:
>> On 15/07/2009 05:01, Ramon F Herrera wrote:
>>
>>> Apparently, in order for my session to proceed faster, I need one of
>>> these two solutions, both implemented on the Windows (client) side:
>>> (1) I run an 'ident' server on port 113
>>> (2) I modify/configure the Windows TCP/IP stack to return RST (Reset)
>>> as soon as it receives the ident request from the server.
>>> Any suggestions on how to achieve any of the two solutions?
>> IIRC, windows rejects connections by default (rather than dropping them)
>> I'm not 100% sure though.
>
>
> > If not, you could add an exception for the ident port which
> > should then cause the firewall to allow it, and then be rejected
> > as nothing is listening.
>
> That is exactly what is happening, Dee. The connection is reject -
> after 30 seconds and 4 packets- because nothing is listening. See
> here:
>
> http://patriot.net/~ramon/misc/rsh-packet-capture.png
>
> I am trying to prevent the 30 second delay.
>
> There is no firewall, BTW, it is a VPN tunnel which nothing blocked.

No it's not, the incoming connections are being dropped, meaning there
is a firewall on that machine (or you have a broken ident server running)
As they are being dropped, there is NO reply to the rsh server so the
only option it has is to wait for a timeout.

To get around this, it MUST be rejected (with a connect failure) by the
firewall or accepted by an application and dealt with.
you should be able to add an exception to the windows firewall for that
port, meaning it then gets rejected rather then silently dropped.

--
Dee Earley (dee.earley@icode.co.uk)
i-Catcher Development Team

iCode Systems

Ramon F Herrera · External Since: Apr 19, 2007 Posts: 30

On Jul 15, 2:50 pm, Maxwell Lol <nos....DeleteThis@com.invalid> wrote:
> Dee Earley <dee.ear....DeleteThis@icode.co.uk> writes:
> > To get around this, it MUST be rejected (with a connect failure) by
> > the firewall or accepted by an application and dealt with.
> > you should be able to add an exception to the windows firewall for
> > that port, meaning it then gets rejected rather then silently dropped.
>

> or disable the ident request on the Linux server.
> (This seems like the easiest solution).

I tried that too, and it is indeed another solution. See below.

The moral of the story is what I have been told repeatedly, and I
concur: Stay away from RSH. Use SSH.

-Ramon The OP

----------------

% cat /etc/xinetd.d/rsh

# default: on
# description: The rshd server is the server for the rcmd(3) routine
and, \
# consequently, for the rsh(1) program. The server provides \
# remote execution facilities with authentication based on \
# privileged port numbers from trusted hosts.

service shell
{
socket_type = stream
wait = no
user = root
log_on_success += USERID <== Change to HOST
log_on_failure += USERID <== Change to HOST
server = /usr/sbin/in.rshd
disable = no
}

Maxwell Lol · External Since: Feb 11, 2009 Posts: 8

Dee Earley <dee.earley DeleteThis @icode.co.uk> writes:

> To get around this, it MUST be rejected (with a connect failure) by
> the firewall or accepted by an application and dealt with.
> you should be able to add an exception to the windows firewall for
> that port, meaning it then gets rejected rather then silently dropped.

or disable the ident request on the Linux client. (This seems like
the easiest solution. Either it's a config issue, or you install ssh,
and rename it rsh, at worst you install a new version from source.)

Some tips here:

http://www.starnet.com/xwin32kb/rsh_or_rexec_sessions_take_a_long_time_to_login/

"Man-wai Chang to The Doo · External Since: Jul 05, 2009 Posts: 9

> C:\> rsh hostname date
>
> Much to my dismay, the above command takes -consistently- 30 seconds
> to execute. This sounds like some sort of "tar pit" behind the scenes.
>
> From Linux to Linux, the command takes 1 second.
>
> How can I investigate exactly what is going on?

Reverse DNS lookup? Ident lookup?

--
@~@ Might, Courage, Vision, SINCERITY.
/ v \ Simplicity is Beauty! May the Force and Farce be with you!
/( _ )\ (Ubuntu 9.04) Linux 2.6.30.1
^ ^ 21:22:01 up 5 days 5:51 1 user load average: 1.35 1.66 1.51
???! ???! ???! ???! ???! ???! ????? (CSSA):
http://www.swd.gov.hk/tc/index/site_pubsvc/page_socsecu/sub_addressesa