Interpreting iowait value from output of iostat

Ask Questions · External Since: Jan 10, 2011 Posts: 1

Hi,

Need some clarifications on my understanding of iostat command.

Pasting the first line of output from this command .

$iostat
Linux 2.6.32-21-generic (desktop) Monday 10 January 2011 _i686_ (2
CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
0.09 0.03 0.74 0.07
0.00 99.08

My confusion is with the %iowait column. The man page says : iostat
"show the percentage of time that the CPU or CPUs were idle during
which the system had an outstanding disk I/O request ".

So , what I understood is that when the CPU is mostly idle and have
free cycles, and there is an IO request , the CPU can immediately
handle it since it has free cycles. Now if the CPU is 100% busy and
has no free cycles to handle an IO request , and there is a IO request
during that time , %iowait value is expected to increase based on my
understanding as the the request is waiting because CPU is busy and
has no free cycles left.

So , if there is an increase in IO wait time , we need to check the
CPU and memory utilization . There might me a possibility of bad
blocks in the disk also .

To simulate this , I tried the following .

I executed dd if=/dev/zero of=/home/test till the disk is saturated.
disk saturation I identified by running the iostat command on another
terminal and looking at the %util column after every 5 secs. %iowait
was mostly fluctuating within 35.00 when the disk was 100% saturated.

After waiting for around 3 minutes , I fired another IO intensive
command using dd . The disk was already 100% busy as per the %util
column. Now since the disk is fully saturated and has no free cycles
for the next dd command , I can see the %iowait went to around 70.00
and was fluctuating between and 50 and 70.

I terminated the second dd command and can see value of %iostat coming
down .

Is my understanding clear . Please clarify

Thanks in Advance

Richard Kettlewell · External Since: Feb 19, 2005 Posts: 189

Ask Questions writes:

> My confusion is with the %iowait column. The man page says : iostat
> "show the percentage of time that the CPU or CPUs were idle during
> which the system had an outstanding disk I/O request ".
>
> So , what I understood is that when the CPU is mostly idle and have
> free cycles, and there is an IO request , the CPU can immediately
> handle it since it has free cycles. Now if the CPU is 100% busy and
> has no free cycles to handle an IO request , and there is a IO request
> during that time , %iowait value is expected to increase based on my
> understanding as the the request is waiting because CPU is busy and
> has no free cycles left.

The CPU does not handle IO requests, it issues them. They are handled
by devices such as hard disks. iowait measures the time the CPU is
forced to spend idle because the only process(es) running are blocked on
IO.

--
http://www.greenend.org.uk/rjk/

Doug Freyburger · External Since: Dec 06, 2010 Posts: 10

Richard Kettlewell wrote:
> Ask Questions writes:
>
>> My confusion is with the %iowait column. The man page says : iostat
>> "show the percentage of time that the CPU or CPUs were idle during
>> which the system had an outstanding disk I/O request ".
>>
>> So , what I understood is that when the CPU is mostly idle and have
>> free cycles, and there is an IO request , the CPU can immediately
>> handle it since it has free cycles. Now if the CPU is 100% busy and
>> has no free cycles to handle an IO request , and there is a IO request
>> during that time , %iowait value is expected to increase based on my
>> understanding as the the request is waiting because CPU is busy and
>> has no free cycles left.
>
> The CPU does not handle IO requests, it issues them. They are handled
> by devices such as hard disks. iowait measures the time the CPU is
> forced to spend idle because the only process(es) running are blocked on
> IO.

Get an old fashioned floppy and run against it. Compare the IOwait
result against the same run on the internal boot drive. The wait for
the floppy will be much higher.

Mount a CIFS share to a slow PC and run against it. Compare the IOwait
result against the same run on the internal boot drive. The wait for
network response will be much higher.

Lately the times I see the highest IOwait percentage is when I use CIFS
mounts to transfer very large Oracle database backups to Windows for
backup. CIFS is chatty to transfer data and the Windows hosts are in no
great hurry to send the next response at any point in the protocol. It
may be easy but it's sure not fast. Running NFS service on the Windows
end is faster but it isn't bundled. Sigh.

Even with 30+% IOwait interactive system response tends to be like
nothing else is running at all. It looked wierd the first few times I
encountered a system with a load average over 10 with no interactive
response lag.