df - disk space - gives incoorect value

John · External Since: May 29, 2009 Posts: 5

Hi

I had reached 100% disk usage due to growth of some log files.

I deleted several large log files but 'df' still says 100% used.

Any idea why?

Regards
John

J G Miller · External Since: Dec 08, 2004 Posts: 34

On Wed, 07 Oct 2009 14:03:54 +0100, John wrote:

> I deleted several large log files but 'df' still says 100% used.

Remember also that df reports the "available" disk space, and that
some disk space is reserved for use only by root. So root can
fill up more of the disk space even when df reports that it is
at 100%.

Hence even if you delete some files, the df usage may not change
from 100%.

Lew Pitcher · External Since: May 16, 2007 Posts: 54

On October 7, 2009 09:03, in alt.os.linux.debian, John (john1949@yahoo.com)
wrote:

> Hi
>
> I had reached 100% disk usage due to growth of some log files.
>
> I deleted several large log files but 'df' still says 100% used.
>
> Any idea why?

I bet you deleted some of the logs that syslogd updates, right?
If so, recycle syslogd (shut it down and restart it).

You see, in Linux, the space used by a file isn't made available until all
hardlinks to that file have been deleted AND all processes that use the
file have closed it. The files used by syslog are held open as long as
syslog is running. Thus, you can delete the fileNAME all you want, but the
FILE won't delete until you shut down syslog.

--
Lew Pitcher

Master Codewright & JOAT-in-training | Registered Linux User #112576
http://pitcher.digitalfreehold.ca/ | GPG public key available by request
---------- Slackware - Because I know what I'm doing. ------

Anton Ertl · External Since: May 21, 2006 Posts: 179

"John" <john1949.RemoveThis@yahoo.com> writes:
>Hi
>
>I had reached 100% disk usage due to growth of some log files.
>
>I deleted several large log files but 'df' still says 100% used.
>
>Any idea why?

The daemons writing to these files still have them open, so these
files are still there. You have to stop these daemons and restart
them. In the future, the way to delete the data without stopping the
daemons is:

echo -n >logfile

- anton
--
M. Anton Ertl Some things have to be seen to be believed
anton.RemoveThis@mips.complang.tuwien.ac.at Most things have to be believed to be seen
http://www.complang.tuwien.ac.at/anton/home.html

John · External Since: May 29, 2009 Posts: 5

"Lew Pitcher" <lpitcher DeleteThis @teksavvy.com> wrote in message
news:8e2aa$4acc95aa$cef8a40f$30022@TEKSAVVY.COM...
> On October 7, 2009 09:03, in alt.os.linux.debian, John
> (john1949@yahoo.com)
> wrote:
>
>> Hi
>>
>> I had reached 100% disk usage due to growth of some log files.
>>
>> I deleted several large log files but 'df' still says 100% used.
>>
>> Any idea why?
>
> I bet you deleted some of the logs that syslogd updates, right?
> If so, recycle syslogd (shut it down and restart it).
>
> You see, in Linux, the space used by a file isn't made available until all
> hardlinks to that file have been deleted AND all processes that use the
> file have closed it. The files used by syslog are held open as long as
> syslog is running. Thus, you can delete the fileNAME all you want, but the
> FILE won't delete until you shut down syslog.
>
>
> --
> Lew Pitcher
>
> Master Codewright & JOAT-in-training | Registered Linux User #112576
> http://pitcher.digitalfreehold.ca/ | GPG public key available by request
> ---------- Slackware - Because I know what I'm doing. ------
>
>

Hi

OK. The only deamon I can find is rsyslog. I'm Debian Lenny.

So I
/etc/init.d/rsyslog stop
then
/etc/init.d/rsyslog start

but the usage is still 100%.

Regards
John

Lew Pitcher · External Since: May 16, 2007 Posts: 54

On October 7, 2009 10:25, in alt.os.linux.debian, Anton Ertl
(anton@mips.complang.tuwien.ac.at) wrote:

> "John" <john1949 DeleteThis @yahoo.com> writes:
>>Hi
>>
>>I had reached 100% disk usage due to growth of some log files.
>>
>>I deleted several large log files but 'df' still says 100% used.
>>
>>Any idea why?
>
> The daemons writing to these files still have them open, so these
> files are still there. You have to stop these daemons and restart
> them. In the future, the way to delete the data without stopping the
> daemons is:
>
> echo -n >logfile

Nope. That's as bad or worse than
rm logfile

The command
echo -n >logfile
simply
a) unlink(2)s the current file named "logfile", and
b) creat(2)s a new file named "logfile"
it does not cause any processes that have the inode for the old "logfile"
open to switch to the inode for the new "logfile".

Instead of seeing that the logfile doesn't exist and freespace hasn't
returned, you see that the logfile exists, is empty, and the
freespace /still/ hasn't returned.

No. The only way to free up filespace allocated to an open file with no
filenames (i.e. a "deleted" file) is to cause all processes that have the
file open to close it.

--
Lew Pitcher

Master Codewright & JOAT-in-training | Registered Linux User #112576
http://pitcher.digitalfreehold.ca/ | GPG public key available by request
---------- Slackware - Because I know what I'm doing. ------

Anton Ertl · External Since: May 21, 2006 Posts: 179

Lew Pitcher <lpitcher DeleteThis @teksavvy.com> writes:
>On October 7, 2009 10:25, in alt.os.linux.debian, Anton Ertl
>(anton@mips.complang.tuwien.ac.at) wrote:
>> In the future, the way to delete the data without stopping the
>> daemons is:
>>
>> echo -n >logfile
>
>Nope. That's as bad or worse than
> rm logfile
>
>The command
> echo -n >logfile
>simply
> a) unlink(2)s the current file named "logfile", and
> b) creat(2)s a new file named "logfile"

[a4:/tmp:41248] echo foo >a
[a4:/tmp:41249] ln a b
[a4:/tmp:41250] echo -n >a
[a4:/tmp:41251] cat b
[a4:/tmp:41252] ls -l a b
-rw-r--r-- 2 anton users 0 Oct 7 17:16 a
-rw-r--r-- 2 anton users 0 Oct 7 17:16 b

a and b are still the same file (note the link count in addition to
size and content), so no unlink() and creat() has happened, just a
truncation.

That's with bash, maybe some other shells implement ">" in the way you
describe. To be sure, one can make the test I made.

>it does not cause any processes that have the inode for the old "logfile"
>open to switch to the inode for the new "logfile".

No, it's still the same file and the same inode, there is just no more
data in it; and of course all the space formerly consumed by that data
is available again.

- anton
--
M. Anton Ertl Some things have to be seen to be believed
anton DeleteThis @mips.complang.tuwien.ac.at Most things have to be believed to be seen
http://www.complang.tuwien.ac.at/anton/home.html

Unruh · External Since: May 27, 2005 Posts: 2213

"John" <john1949.DeleteThis@yahoo.com> writes:

>Hi

>I had reached 100% disk usage due to growth of some log files.

>I deleted several large log files but 'df' still says 100% used.

>Any idea why?

Yes. You deleted an open file. While the file name is deleted, so
nothing else can reference the file, the program ( eg syslogd) that has
the file open refers to it by inode, and that is still open. What you
need to do is to restart all programs that have those files open after
deleting them. If you look into /etc/logrotate.d you will see things
like (/etc/logrotate.d/syslog)

postrotate
[ -f /var/run/syslog-ng.pid ] && kill -HUP `cat
/var/run/syslog-ng.pid` || true
[ -f /var/run/syslogd.pid ] && kill -HUP `cat
/var/run/syslogd.pid` || true

which kills with a a signal that tells syslogd to shut and reopen any
log file it has open. That is to release the inodes.

>Regards
>John

Unruh · External Since: May 27, 2005 Posts: 2213

"John" <john1949 RemoveThis @yahoo.com> writes:

>"Lew Pitcher" <lpitcher RemoveThis @teksavvy.com> wrote in message
>news:8e2aa$4acc95aa$cef8a40f$30022@TEKSAVVY.COM...
>> On October 7, 2009 09:03, in alt.os.linux.debian, John
>> (john1949@yahoo.com)
>> wrote:
>>
>>> Hi
>>>
>>> I had reached 100% disk usage due to growth of some log files.
>>>
>>> I deleted several large log files but 'df' still says 100% used.
>>>
>>> Any idea why?
>>
>> I bet you deleted some of the logs that syslogd updates, right?
>> If so, recycle syslogd (shut it down and restart it).
>>
>> You see, in Linux, the space used by a file isn't made available until all
>> hardlinks to that file have been deleted AND all processes that use the
>> file have closed it. The files used by syslog are held open as long as
>> syslog is running. Thus, you can delete the fileNAME all you want, but the
>> FILE won't delete until you shut down syslog.
>>
>>
>> --
>> Lew Pitcher
>>
>> Master Codewright & JOAT-in-training | Registered Linux User #112576
>> http://pitcher.digitalfreehold.ca/ | GPG public key available by request
>> ---------- Slackware - Because I know what I'm doing. ------
>>
>>

>Hi

>OK. The only deamon I can find is rsyslog. I'm Debian Lenny.

>So I
>/etc/init.d/rsyslog stop
>then
>/etc/init.d/rsyslog start

>but the usage is still 100%.

Then you have to give us more infomation. Which files were they that
had huge amounts of data in them that you deleted?
Note, rsyslog is NOT the only daemon you have running.

John · External Since: May 29, 2009 Posts: 5

"Unruh" <unruh-spam DeleteThis @physics.ubc.ca> wrote in message
news:XO4zm.47969$PH1.21329@edtnps82...
> "John" <john1949 DeleteThis @yahoo.com> writes:
>
>
>>"Lew Pitcher" <lpitcher DeleteThis @teksavvy.com> wrote in message
>>news:8e2aa$4acc95aa$cef8a40f$30022@TEKSAVVY.COM...
>>> On October 7, 2009 09:03, in alt.os.linux.debian, John
>>> (john1949@yahoo.com)
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> I had reached 100% disk usage due to growth of some log files.
>>>>
>>>> I deleted several large log files but 'df' still says 100% used.
>>>>
>>>> Any idea why?
>>>
>>> I bet you deleted some of the logs that syslogd updates, right?
>>> If so, recycle syslogd (shut it down and restart it).
>>>
>>> You see, in Linux, the space used by a file isn't made available until
>>> all
>>> hardlinks to that file have been deleted AND all processes that use the
>>> file have closed it. The files used by syslog are held open as long as
>>> syslog is running. Thus, you can delete the fileNAME all you want, but
>>> the
>>> FILE won't delete until you shut down syslog.
>>>
>>>
>>> --
>>> Lew Pitcher
>>>
>>> Master Codewright & JOAT-in-training | Registered Linux User #112576
>>> http://pitcher.digitalfreehold.ca/ | GPG public key available by
>>> request
>>> ---------- Slackware - Because I know what I'm
>>> ------
>>>
>>>
>
>
>>Hi
>
>>OK. The only deamon I can find is rsyslog. I'm Debian Lenny.
>
>>So I
>>/etc/init.d/rsyslog stop
>>then
>>/etc/init.d/rsyslog start
>
>>but the usage is still 100%.
>
>
> Then you have to give us more infomation. Which files were they that
> had huge amounts of data in them that you deleted?
> Note, rsyslog is NOT the only daemon you have running.
>

I'm guessing it was log files in var/log/mysql and var/log/apache2.

Regards
John

Chris Davies · External Since: Apr 13, 2004 Posts: 284

Lew Pitcher <lpitcher.TakeThisOut@teksavvy.com> wrote:
> echo -n >logfile
> simply
> a) unlink(2)s the current file named "logfile", and
> b) creat(2)s a new file named "logfile"

It doesn't in any shell I've used. The ">" operation is equivalent to
"truncate or create", just like the default action of the underlying
creat(2) system call:

creat() is equivalent to open() with flags equal to
O_CREAT|O_WRONLY|O_TRUNC.

There is no unlink(2) involved.
Chris

Unruh · External Since: May 27, 2005 Posts: 2213

"John" <john1949.TakeThisOut@yahoo.com> writes:

>"Unruh" <unruh-spam.TakeThisOut@physics.ubc.ca> wrote in message
>news:XO4zm.47969$PH1.21329@edtnps82...
>> "John" <john1949.TakeThisOut@yahoo.com> writes:
>>
>>
>>>"Lew Pitcher" <lpitcher.TakeThisOut@teksavvy.com> wrote in message
>>>news:8e2aa$4acc95aa$cef8a40f$30022@TEKSAVVY.COM...
>>>> On October 7, 2009 09:03, in alt.os.linux.debian, John
>>>> (john1949@yahoo.com)
>>>> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I had reached 100% disk usage due to growth of some log files.
>>>>>
>>>>> I deleted several large log files but 'df' still says 100% used.
>>>>>
>>>>> Any idea why?
>>>>
>>>> I bet you deleted some of the logs that syslogd updates, right?
>>>> If so, recycle syslogd (shut it down and restart it).
>>>>
>>>> You see, in Linux, the space used by a file isn't made available until
>>>> all
>>>> hardlinks to that file have been deleted AND all processes that use the
>>>> file have closed it. The files used by syslog are held open as long as
>>>> syslog is running. Thus, you can delete the fileNAME all you want, but
>>>> the
>>>> FILE won't delete until you shut down syslog.
>>>>
>>>>
>>>> --
>>>> Lew Pitcher
>>>>
>>>> Master Codewright & JOAT-in-training | Registered Linux User #112576
>>>> http://pitcher.digitalfreehold.ca/ | GPG public key available by
>>>> request
>>>> ---------- Slackware - Because I know what I'm
>>>> ------
>>>>
>>>>
>>
>>
>>>Hi
>>
>>>OK. The only deamon I can find is rsyslog. I'm Debian Lenny.
>>
>>>So I
>>>/etc/init.d/rsyslog stop
>>>then
>>>/etc/init.d/rsyslog start
>>
>>>but the usage is still 100%.
>>
>>
>> Then you have to give us more infomation. Which files were they that
>> had huge amounts of data in them that you deleted?
>> Note, rsyslog is NOT the only daemon you have running.
>>

>I'm guessing it was log files in var/log/mysql and var/log/apache2.

Then try closing down and restarting mysql and apache.
service httpd restart
service mysqld restart
They both probably keep open their log files when running as well.

Note both httpd and mysqld are daemons.

John · External Since: May 29, 2009 Posts: 5

"Unruh" <unruh-spam.DeleteThis@physics.ubc.ca> wrote in message
news:gD7zm.47983$PH1.25817@edtnps82...
> "John" <john1949.DeleteThis@yahoo.com> writes:
>
>
>>"Unruh" <unruh-spam.DeleteThis@physics.ubc.ca> wrote in message
>>news:XO4zm.47969$PH1.21329@edtnps82...
>>> "John" <john1949.DeleteThis@yahoo.com> writes:
>>>
>>>
>>>>"Lew Pitcher" <lpitcher.DeleteThis@teksavvy.com> wrote in message
>>>>news:8e2aa$4acc95aa$cef8a40f$30022@TEKSAVVY.COM...
>>>>> On October 7, 2009 09:03, in alt.os.linux.debian, John
>>>>> (john1949@yahoo.com)
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I had reached 100% disk usage due to growth of some log files.
>>>>>>
>>>>>> I deleted several large log files but 'df' still says 100% used.
>>>>>>
>>>>>> Any idea why?
>>>>>
>>>>> I bet you deleted some of the logs that syslogd updates, right?
>>>>> If so, recycle syslogd (shut it down and restart it).
>>>>>
>>>>> You see, in Linux, the space used by a file isn't made available until
>>>>> all
>>>>> hardlinks to that file have been deleted AND all processes that use
>>>>> the
>>>>> file have closed it. The files used by syslog are held open as long as
>>>>> syslog is running. Thus, you can delete the fileNAME all you want, but
>>>>> the
>>>>> FILE won't delete until you shut down syslog.
>>>>>
>>>>>
>>>>> --
>>>>> Lew Pitcher
>>>>>
>>>>> Master Codewright & JOAT-in-training | Registered Linux User #112576
>>>>> http://pitcher.digitalfreehold.ca/ | GPG public key available by
>>>>> request
>>>>> ---------- Slackware - Because I know what I'm
>>>>> ------
>>>>>
>>>>>
>>>
>>>
>>>>Hi
>>>
>>>>OK. The only deamon I can find is rsyslog. I'm Debian Lenny.
>>>
>>>>So I
>>>>/etc/init.d/rsyslog stop
>>>>then
>>>>/etc/init.d/rsyslog start
>>>
>>>>but the usage is still 100%.
>>>
>>>
>>> Then you have to give us more infomation. Which files were they that
>>> had huge amounts of data in them that you deleted?
>>> Note, rsyslog is NOT the only daemon you have running.
>>>
>
>>I'm guessing it was log files in var/log/mysql and var/log/apache2.
>
> Then try closing down and restarting mysql and apache.
> service httpd restart
> service mysqld restart
> They both probably keep open their log files when running as well.
>
>
> Note both httpd and mysqld are daemons.

Many thanks. That was the answer..

/etc/init.d/apache2 stop | start
/etc/init.d/mysql stop | start

and my servers is up again.

I was focusing on the syslog and of course there are other daemons as you
said.
Even when I replied that I had deleted the apache2 and mysql logs, I still
didn't make the mental connection.

I guess next time I need to keep the inode linked to the same file by using
cp /dev/null /var/llog/error_log

I can now get back to work again.

Regards
John