modifying memory usage of a program (F-Prot)?

news · External Since: Aug 15, 2005 Posts: 5

It started with my noticing our fileserver closing all open connections
and restarting HTTPD and other processes each night.
According to the /var/log/messages, I'd been getting a slew of these
messages the same time each night:
Aug 15 05:03:21 fileserve kernel: Out of Memory: Killed process 30974
(httpd).
Aug 15 05:03:27 fileserve kernel: Out of Memory: Killed process 30975
(httpd).
Aug 15 05:03:32 fileserve kernel: Out of Memory: Killed process 7835
(sshd).
But there's nothing set to run at 5am each day.

With the help from the newsgroup, I set up a cron to dump "top" for a
file every 20 minutes, and discovered the F-prot anti-virus program
which starts it's check at 4am each morning, starting out taking up
very low memory and then more and more over time. Less and less free
swap and buffers, until finally an hour later, "Out of Memory" errors
and things start shutting down.

The fileserver has 512MB of RAM and 256MB of swap. (That's what Fedora
Core2's autoDisk Druid suggested to make it.) I know I can renice a
process to give it more processor priority...is there some way I can
tell F-prot to try to run using less memory? I really don't want to go
without a daily virus check, but this crashing of the server every
night is not a good thing either.

Thanks for any suggestions!
Liam

Malke · External Since: Jul 13, 2005 Posts: 15

news.RemoveThis@celticbear.com wrote:

> It started with my noticing our fileserver closing all open
> connections and restarting HTTPD and other processes each night.
> According to the /var/log/messages, I'd been getting a slew of these
> messages the same time each night:
> Aug 15 05:03:21 fileserve kernel: Out of Memory: Killed process 30974
> (httpd).
> Aug 15 05:03:27 fileserve kernel: Out of Memory: Killed process 30975
> (httpd).
> Aug 15 05:03:32 fileserve kernel: Out of Memory: Killed process 7835
> (sshd).
> But there's nothing set to run at 5am each day.
>
> With the help from the newsgroup, I set up a cron to dump "top" for a
> file every 20 minutes, and discovered the F-prot anti-virus program
> which starts it's check at 4am each morning, starting out taking up
> very low memory and then more and more over time. Less and less free
> swap and buffers, until finally an hour later, "Out of Memory" errors
> and things start shutting down.
>
> The fileserver has 512MB of RAM and 256MB of swap. (That's what Fedora
> Core2's autoDisk Druid suggested to make it.) I know I can renice a
> process to give it more processor priority...is there some way I can
> tell F-prot to try to run using less memory? I really don't want to go
> without a daily virus check, but this crashing of the server every
> night is not a good thing either.
>
> Thanks for any suggestions!
> Liam

Have you tried contacting F-Prot's tech support? If this is a known
issue, they would have information about workarounds, settings, etc.
I've run F-Prot on SuSE 9.2 acting as a server for some Windows boxen
and I use it on all my Windows machines with no problem. Do you have
the latest version of their software?

Malke
--
"I have a cunning plan..."

news · External Since: Aug 15, 2005 Posts: 5

John-Paul Stewart wrote:
> news.DeleteThis@celticbear.com wrote:
....is there some way I can
> > tell F-prot to try to run using less memory? I really don't want to go
> > without a daily virus check, but this crashing of the server every
> > night is not a good thing either.
>
> Are you running a recent version of F-Prot? ISTR there being a problem
> with a version leaking memory a couple of years ago. (I switched to

Program version: 4.5.4
Engine version: 3.16.6

I think the latest is 4.6. I'm downloading it now.
I'm using the same version on two machines, the fileserver and my home
PC. The home PC hasn't had any problems, is also Fedora Core, and has
512MB RAM, but 1.5GB of swap! (Auto Disk Druid as well. Weird.)
So, I'm going to try....

> As a quick fix, you could create a swap *file* to add extra swap space.
> Find a filesystem with more than 1GB free space and 'cd' to a
> convenient directory on it and execute (as root):
>
> dd if=/dev/zero of=swapfile bs=1M count=1K
> mkswap swapfile
> swapon swapfile
>

Thanks for the suggestion!
BTW, I noticed once I did the "dd" line, my command prompt didn't
return. Just carriage returned. So I did the other two lines, and then
CTRL-C to return to the command prompt. Seemed to work OK, as I now
have a "swapfile" file that's 600MB large. But I MAN dd and didn't see
anything about entering some sub-shell when using the dd command. Did
CTRL-C mess anything up, you think?

Also, while it looks like I may need to keep the swapfile file, if I
want to deactivate it with swapoff, will that be all I need to type?
And it'll go back to using the original "partition"? I see "swaoff -a"
would be bad!
"swapoff swapfile" perhaps? I see the original swap in my /etc/fstab,
so if I just did "swapoff", that should work OK?

(Sorry for the inane questions, but I've never worked with the swapfile
before, and don't want to make some stupid mistake.)

Thanks!
Liam

John-Paul Stewart · External Since: Aug 15, 2005 Posts: 4

news RemoveThis @celticbear.com wrote:
> It started with my noticing our fileserver closing all open connections
> and restarting HTTPD and other processes each night.
> According to the /var/log/messages, I'd been getting a slew of these
> messages the same time each night:
> Aug 15 05:03:21 fileserve kernel: Out of Memory: Killed process 30974
> (httpd).
> Aug 15 05:03:27 fileserve kernel: Out of Memory: Killed process 30975
> (httpd).
> Aug 15 05:03:32 fileserve kernel: Out of Memory: Killed process 7835
> (sshd).
> But there's nothing set to run at 5am each day.
>
> With the help from the newsgroup, I set up a cron to dump "top" for a
> file every 20 minutes, and discovered the F-prot anti-virus program
> which starts it's check at 4am each morning, starting out taking up
> very low memory and then more and more over time. Less and less free
> swap and buffers, until finally an hour later, "Out of Memory" errors
> and things start shutting down.
>
> The fileserver has 512MB of RAM and 256MB of swap. (That's what Fedora
> Core2's autoDisk Druid suggested to make it.) I know I can renice a
> process to give it more processor priority...is there some way I can
> tell F-prot to try to run using less memory? I really don't want to go
> without a daily virus check, but this crashing of the server every
> night is not a good thing either.

Are you running a recent version of F-Prot? ISTR there being a problem
with a version leaking memory a couple of years ago. (I switched to
ClamAV though, so I'm not sure if it got fixed. I'm also not entirely
sure it was F-Prot that had the problem in the first place.) With the
buggy software, it didn't matter how much RAM you had; eventually the
memory leak would cause all of it to be used.

Also, check with the makers of F-Prot for support. You should be
entitled to support with your paid license.

As a quick fix, you could create a swap *file* to add extra swap space.
Find a filesystem with more than 1GB free space and 'cd' to a
convenient directory on it and execute (as root):

dd if=/dev/zero of=swapfile bs=1M count=1K
mkswap swapfile
swapon swapfile

I suspect that you'll still see the problem occur. It'll just take
longer to fill the extra swap space. (Use the 'swapoff' command to
disable the swap file, and then delete it when you're done with the
test.) However, if it doesn't happen with the extra swap file, then
you'll know that your machine was really just memory starved rather than
there being a memory leak in F-Prot. Make the extra swap space
permanent and/or add more RAM to the machine.

John-Paul Stewart · External Since: Aug 15, 2005 Posts: 4

news.TakeThisOut@celticbear.com wrote:
> John-Paul Stewart wrote:
>
>>As a quick fix, you could create a swap *file* to add extra swap space.
>> Find a filesystem with more than 1GB free space and 'cd' to a
>>convenient directory on it and execute (as root):
>>
>>dd if=/dev/zero of=swapfile bs=1M count=1K
>>mkswap swapfile
>>swapon swapfile
>>
>
>
> Thanks for the suggestion!
> BTW, I noticed once I did the "dd" line, my command prompt didn't
> return. Just carriage returned.

You didn't wait long enough. Writing a gigabyte of zeroes takes a while.

> So I did the other two lines, and then
> CTRL-C to return to the command prompt. Seemed to work OK, as I now
> have a "swapfile" file that's 600MB large. But I MAN dd and didn't see
> anything about entering some sub-shell when using the dd command. Did
> CTRL-C mess anything up, you think?

It just ended prematurely with 600MB instead of the 1GB had the command
completed.

> Also, while it looks like I may need to keep the swapfile file, if I
> want to deactivate it with swapoff, will that be all I need to type?
> And it'll go back to using the original "partition"? I see "swaoff -a"
> would be bad!

Indeed!

> "swapoff swapfile" perhaps?

I guess. 'man swapoff' should tell you...I haven't had to do that in a
while so I don't remember and I'm too lazy to look it up.

news · External Since: Aug 15, 2005 Posts: 5

John-Paul Stewart wrote:
> news.RemoveThis@celticbear.com wrote:
> > John-Paul Stewart wrote:
> >
> >>As a quick fix, you could create a swap *file* to add extra swap space.
> >> Find a filesystem with more than 1GB free space and 'cd' to a
> >>convenient directory on it and execute (as root):
> >>
> >>dd if=/dev/zero of=swapfile bs=1M count=1K
> >>mkswap swapfile
> >>swapon swapfile
> >>
> >
> >
> > Thanks for the suggestion!
> > BTW, I noticed once I did the "dd" line, my command prompt didn't
> > return. Just carriage returned.
>
> You didn't wait long enough. Writing a gigabyte of zeroes takes a while.
>
Ah, yeah, I see. Warning to any other newbies reading this...be
patient! =)
I upgraded the software yesterday, still crashed last night. The "top"
output shows it ran out of buffers and swap...which was indeed the same
size is before.
So I deleted and followed your instructions again...properly, and
voila! The "top" now shows swap being 1GB larger, and "swapon -s" shows
that new file as part of the swap.
What a cool trick that is. =)
So I anticipate everything working OK tonight.
Thanks for the help!!
Liam

news · External Since: Aug 15, 2005 Posts: 5

Chiefy wrote:
> 15 Aug 2005 14:48 UTC, news.RemoveThis@celticbear.com typed:
> > It started with my noticing our fileserver closing all open connections
> > and restarting HTTPD and other processes each night.
> > According to the /var/log/messages, I'd been getting a slew of these
> > messages the same time each night:
> > Aug 15 05:03:21 fileserve kernel: Out of Memory: Killed process 30974
> > (httpd).
> > Aug 15 05:03:27 fileserve kernel: Out of Memory: Killed process 30975
> > (httpd).
> > Aug 15 05:03:32 fileserve kernel: Out of Memory: Killed process 7835
> > (sshd).
> > But there's nothing set to run at 5am each day.
>
> Wonder if you have an OOM killer enabled. Turning it off won't help you
> find the problem, but it should stop other processes being killed off by
> your rogue 'possibly' program.
>
I have no clue. It's Fedora Core 2, with as few of the built-in
packages as I could get away with. (I may have built it just using the
"Server" setup path.) So, I don't even know where to start looking to
see if it has an OOM killer.
/var/log/messages doesn't give any indication WHAT is closing the
services when it runs out of memory, they just are.

Liam

John-Paul Stewart · External Since: Aug 15, 2005 Posts: 4

news DeleteThis @celticbear.com wrote:
> John-Paul Stewart wrote:
>
>>news@celticbear.com wrote:
>>
>>>John-Paul Stewart wrote:
>>>
>>>
>>>>As a quick fix, you could create a swap *file* to add extra swap space.
>>>> Find a filesystem with more than 1GB free space and 'cd' to a
>>>>convenient directory on it and execute (as root):
>>>>
>>>>dd if=/dev/zero of=swapfile bs=1M count=1K
>>>>mkswap swapfile
>>>>swapon swapfile
>>>>
>>>
>>>
>>>Thanks for the suggestion!
>>>BTW, I noticed once I did the "dd" line, my command prompt didn't
>>>return. Just carriage returned.
>>
>>You didn't wait long enough. Writing a gigabyte of zeroes takes a while.
>>
>
> Ah, yeah, I see. Warning to any other newbies reading this...be
> patient! =)

I really should have mentioned that in my first post. I've done it so
often that I tend to forget others may not have created such a large
file with dd before and thus may not know that it will take a while.

> I upgraded the software yesterday, still crashed last night. The "top"
> output shows it ran out of buffers and swap...which was indeed the same
> size is before.
> So I deleted and followed your instructions again...properly, and
> voila! The "top" now shows swap being 1GB larger, and "swapon -s" shows
> that new file as part of the swap.
> What a cool trick that is. =)
> So I anticipate everything working OK tonight.
> Thanks for the help!!

It'll be interesting to see if the problem re-occurs, but later due to
the larger available swap space (indicating a memory leak in the app) or
if it goes away (indicating your system was just starved for memory in
which case you may want to add more RAM as well as the swap space.)

Unruh · External Since: Aug 16, 2005 Posts: 5

John-Paul Stewart <jpstewart.DeleteThis@binaryfoundry.ca> writes:

>> I upgraded the software yesterday, still crashed last night. The "top"
>> output shows it ran out of buffers and swap...which was indeed the same
>> size is before.
>> So I deleted and followed your instructions again...properly, and
>> voila! The "top" now shows swap being 1GB larger, and "swapon -s" shows
>> that new file as part of the swap.
>> What a cool trick that is. =)
>> So I anticipate everything working OK tonight.
>> Thanks for the help!!

>It'll be interesting to see if the problem re-occurs, but later due to
>the larger available swap space (indicating a memory leak in the app) or
>if it goes away (indicating your system was just starved for memory in
>which case you may want to add more RAM as well as the swap space.)

Yee gads. Get 1GB ofmemory and add that. swap is a hopeless way of creating
memory. It is about a factor of 1000 slower than memory. If you program
starts swappping you might as well forget about it. If you run out of swap,
you should definitley forget about it. Your machine will spend all of its
time swapping and none doing anything useful.
GET MORE MEMORY. Do not increase swap.it is useless for accomplishing
anything.

John-Paul Stewart · External Since: Aug 15, 2005 Posts: 4

Unruh wrote:
> John-Paul Stewart <jpstewart RemoveThis @binaryfoundry.ca> writes:
>
>
>
>>>I upgraded the software yesterday, still crashed last night. The "top"
>>>output shows it ran out of buffers and swap...which was indeed the same
>>>size is before.
>>>So I deleted and followed your instructions again...properly, and
>>>voila! The "top" now shows swap being 1GB larger, and "swapon -s" shows
>>>that new file as part of the swap.
>>>What a cool trick that is. =)
>>>So I anticipate everything working OK tonight.
>>>Thanks for the help!!
>
>
>>It'll be interesting to see if the problem re-occurs, but later due to
>>the larger available swap space (indicating a memory leak in the app) or
>>if it goes away (indicating your system was just starved for memory in
>>which case you may want to add more RAM as well as the swap space.)
>
>
> Yee gads. Get 1GB ofmemory and add that. swap is a hopeless way of creating
> memory. It is about a factor of 1000 slower than memory. If you program
> starts swappping you might as well forget about it. If you run out of swap,
> you should definitley forget about it. Your machine will spend all of its
> time swapping and none doing anything useful.
> GET MORE MEMORY. Do not increase swap.it is useless for accomplishing
> anything.

If you look up thread, I was suggesting the OP add the swap as a
diagnostic tool. If he increases swap by 1GB and the original problem
(F-Prot using all RAM and swap) persists, that's a likely indication of
a memory leak or other problem with the application. In that case, no
amount of additional RAM will help. So why spend the money on extra RAM
until you know for sure that it will help?

However, if the problem does go away with the additonal swap, then I
fully agree that extra RAM is a better long-term solution.

Chiefy · External Since: Aug 16, 2005 Posts: 1

15 Aug 2005 14:48 UTC, news RemoveThis @celticbear.com typed:
> It started with my noticing our fileserver closing all open connections
> and restarting HTTPD and other processes each night.
> According to the /var/log/messages, I'd been getting a slew of these
> messages the same time each night:
> Aug 15 05:03:21 fileserve kernel: Out of Memory: Killed process 30974
> (httpd).
> Aug 15 05:03:27 fileserve kernel: Out of Memory: Killed process 30975
> (httpd).
> Aug 15 05:03:32 fileserve kernel: Out of Memory: Killed process 7835
> (sshd).
> But there's nothing set to run at 5am each day.

Wonder if you have an OOM killer enabled. Turning it off won't help you
find the problem, but it should stop other processes being killed off by
your rogue 'possibly' program.

--
Meader's Law:

Whatever happens to you, it will previously have happened to everyone you
know, only more so.

Michael Heiming · External Since: Aug 16, 2005 Posts: 1

In comp.os.linux.misc Unruh <unruh-spam RemoveThis @physics.ubc.ca>:
> John-Paul Stewart <jpstewart RemoveThis @binaryfoundry.ca> writes:

>>> I upgraded the software yesterday, still crashed last night.
>>> The "top" output shows it ran out of buffers and swap...which
>>> was indeed the same size is before. So I deleted and
>>> followed your instructions again...properly, and voila! The
>>> "top" now shows swap being 1GB larger, and "swapon -s" shows
>>> that new file as part of the swap. What a cool trick that
>>> is. =) So I anticipate everything working OK tonight. Thanks
>>> for the help!!

>> It'll be interesting to see if the problem re-occurs, but later
>> due to the larger available swap space (indicating a memory
>> leak in the app) or if it goes away (indicating your system was
>> just starved for memory in which case you may want to add more
>> RAM as well as the swap space.)

> Yee gads. Get 1GB ofmemory and add that. swap is a hopeless way
> of creating memory. It is about a factor of 1000 slower than
> memory. If you program starts swappping you might as well
> forget about it. If you run out of swap, you should definitley
> forget about it. Your machine will spend all of its time
> swapping and none doing anything useful. GET MORE MEMORY. Do
> not increase swap.it is useless for accomplishing anything.

Yep, more memory does certainly help and it's cheap enough.
However, I'd enable overcommit memory at first and see what that
does, some apps tend to allocate memory in a strange way.

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo zvpunry RemoveThis @urvzvat.qr | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 348: We're on Token Ring, and it looks like the
token got loose.

news · External Since: Aug 15, 2005 Posts: 5

John-Paul Stewart wrote:
>
> If you look up thread, I was suggesting the OP add the swap as a
> diagnostic tool. If he increases swap by 1GB and the original problem
> (F-Prot using all RAM and swap) persists, that's a likely indication of
> a memory leak or other problem with the application. In that case, no
> amount of additional RAM will help. So why spend the money on extra RAM
> until you know for sure that it will help?
>
> However, if the problem does go away with the additonal swap, then I
> fully agree that extra RAM is a better long-term solution.

Welp, it worked. I dumped "top" at 15 minute intervals, and at times
the used swap space was past what I used to have, but never came close
to all of the new swap space.
So the problem is indeed not the program, but the amount of RAM.
Thanks for the help!! It's greatly appreciated!
(I am definitely writing this trick down in my "cool tips" journal.)
Liam