sluggish system responsiveness under higher IO load

Jens Axboe · External Since: Oct 04, 2006 Posts: 673

On Mon, Oct 09 2006, Jesper Juhl wrote:
> On 08/10/06, Jens Axboe <jens.axboe.DeleteThis@oracle.com> wrote:
> >On Sun, Oct 08 2006, Christian wrote:
> >> Am Freitag, 6. Oktober 2006 17:58 schrieb Paolo Ornati:
> >> > On Sun, 13 Aug 2006 18:15:12 +0200
> >> >
> >> > Matthias Dahl <mlkernel.DeleteThis@mortal-soul.de> wrote:
> >> > > Just let me know once you got them, so I can safely delete them
> >again.
> >> > >
> >> > > At the moment, I am trying without preemption but for example doing a
> >> > > untar kernel sources still results in sluggish system
> >responsiveness. Sad

> >> >
> >> > I used to have this type of problem and 2.6.19-rc1 looks much better
> >> > than 2.6.18.
> >> >
> >> > I'm using CONFIG_PREEMPT + CONFIG_PREEMPT_BKL, CFQ i/o scheduler
> >> > and /proc/sys/vm/swappiness = 20.
> >>
> >>
> >> Which change in the new kernel has made it better? I was following the
> >lkml
> >> very close and didn't see any change that could have fixed that problem.
> >
> >There is a substantial CFQ update, so it could be that. Or it could be
> >something unrelated of course, I didn't check if eg the cpu scheduler
> >changed much. Or vm Smile

> >
> >--
> >Jens Axboe
> >
> I want to chime in here and let you know that I've experienced
> something similar.
>
> I'm using CFQ as my default I/O schedular.
> Since 2.6.18-git<something_I'm_not_sure_of> I've experienced that when
> doing heavy (or even not so heave) disk I/O my system gets very
> sluggish. Observable by the fact that my mouse pointer in X "jumps"
> which it never did before, and switching windows I can see the new
> window repaint slowly whereas earlier it would just snap onto the
> screen.

You should verify that this is CFQ related by booting into the same
kernel with elevator=deadline (or anticipatory) and attempt to reproduce
it, then reboot with CFQ and make sure it reproduces again.

Having profiles of the sluggish period would also be very interesting,
regardless of the outcome of the above test.

--
Jens Axboe

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Matthias Dahl · External Since: Aug 06, 2006 Posts: 10

On Friday 06 October 2006 17:58, Paolo Ornati wrote:

> I used to have this type of problem and 2.6.19-rc1 looks much better
> than 2.6.18.
>
> I'm using CONFIG_PREEMPT + CONFIG_PREEMPT_BKL, CFQ i/o scheduler
> and /proc/sys/vm/swappiness = 20.

I will give 2.6.19 a test in a few weeks when the dust of all the changes have
settled a bit. Smile

As my Mike Galbraith suggested, I made some tests with renicing the IO
intensive applications. This indeed makes a hell of a difference. Currently I
am renicing everything that causes a lot of disk IO to a nice of 19. Even
though this doesn't fix it completely, the occasional short hangs have become
less common.

Unfortunately things have gotten a bit worse again since I switched my machine
over to a linux software RAID5 (3 sata disks attached to the nforce4 ultra)
even though I hoped quite the opposite would happen.

Have a nice weekend,
matthew.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Paolo Ornati · External Since: Apr 13, 2006 Posts: 74

On Sat, 14 Oct 2006 16:39:57 +0200
Matthias Dahl <mlkernel DeleteThis @mortal-soul.de> wrote:

> I will give 2.6.19 a test in a few weeks when the dust of all the changes have
> settled a bit. Smile

-rc2 is rock solid here, but if you want to wait...

>
> As my Mike Galbraith suggested, I made some tests with renicing the IO
> intensive applications. This indeed makes a hell of a difference. Currently I
> am renicing everything that causes a lot of disk IO to a nice of 19. Even
> though this doesn't fix it completely, the occasional short hangs have become
> less common.

Renicing to avoid sluggish system with a simple "cp" or untar is at
best a workaround... so I say: go with .19 and see what happens !

Smile

--
Paolo Ornati
Linux 2.6.19-rc2 on x86_64
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Mike Galbraith · External Since: May 26, 2006 Posts: 368

Greetings,

On Sat, 2006-10-14 at 16:39 +0200, Matthias Dahl wrote:
> On Friday 06 October 2006 17:58, Paolo Ornati wrote:
>
> > I used to have this type of problem and 2.6.19-rc1 looks much better
> > than 2.6.18.
> >
> > I'm using CONFIG_PREEMPT + CONFIG_PREEMPT_BKL, CFQ i/o scheduler
> > and /proc/sys/vm/swappiness = 20.
>
> I will give 2.6.19 a test in a few weeks when the dust of all the changes have
> settled a bit. Smile

>
> As my Mike Galbraith suggested, I made some tests with renicing the IO
> intensive applications. This indeed makes a hell of a difference. Currently I
> am renicing everything that causes a lot of disk IO to a nice of 19. Even
> though this doesn't fix it completely, the occasional short hangs have become
> less common.

(I probably should have been more verbose in my suggestion;)

What I actually suggested was that you try renicing the application you
were experiencing sluggishness with to -10, and retry your IO
interference test to see if you were experiencing scheduling latency or
something else. For example, if your GL application is using lots of
cpu, it will likely not be classified as interactive, and can end up in
the expired array, at which time an IO task can do a long burst of heavy
cpu usage at interactive status, and keep your application off of the
cpu for quite a while. The intent of renicing your application to -10
was to keep it at interactive status, and above the heavy IO tasks. (if
it sleeps at all that should work. there are other scenarios too, but
less likely than this one)

If running IO at nice 19 more or less fixes your problem, I think we can
assume that you are having scheduling troubles, so the thing to do is to
grab some top snapshots showing cpu distribution during a problem
period.

-Mike

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/