2.6.18-rt1

Bill Huey · External Since: Jul 24, 2006 Posts: 84

On Thu, Sep 21, 2006 at 09:56:33AM +0200, Ingo Molnar wrote:
> * Bill Huey <billh.TakeThisOut@gnuppy.monkey.org> wrote:
>
> > [...] If the upstream kernel used RCU function in a task allocation or
> > task struct reading in the first place then call_rcu() would be a
> > clear choice. However, I didn't see it used in that way (I could be
> > wrong) [...]
>
> it was RCU-ified briefly but then it was further improved to direct
> freeing, because upstream _can_ free it directly.

Unfortunately, this is a problem with -rt patch and the lock ordering
in this system when you have to call a memory allocator within an atomic
critical section. I fully accept this as part of what goes into making a
kernel preemptive and I'm ok with it. Not many folks know about the
special case locking rules in the -rt kernel so this might be new to
various folks.

If you're looking for validation of this technique from me and an ego
stroking, then you have it from me. Smile

Fortunately, it's in a non-critical place so this should *not* be too
much of a problem, but I've already encountered oddies trying to
allocate a pool of entities for populating a free list under an atomic
critical section of some sort for some code I've been writing. This is
a significant problem with kernel coding in -rt, but I can't say what
the general solution is other than making the memory allocators
non-preemptible by reverting the locks back to raw spinlocks, etc...
using lock-break, who knows. I'm ok with the current scenario, but this
could eventually be a larger problem.

bill

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar · External Since: May 15, 2006 Posts: 2457

* Deepak Saxena <dsaxena.RemoveThis@plexity.net> wrote:

> I am seeing an intermittent lock up on the ARM Versatile board during
> the ALSA driver init that only shows up with (PREEMPT_RT &
> !HIGH_RES_TIMERS & ARM_EABI) enabled. If HRT is disabled and EABI is
> enabled, the kernel works every time, and same with !RT & !HRT & EABI.
> I get no oops, just a complete lock up with no console output.

does enabling LOCKDEP give you any better info? (It might not make a
difference on the bootup that locks, but maybe you'll get a lockdep clue
about the problem in one of the successful bootups.)

Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Deepak Saxena · External Since: May 16, 2006 Posts: 29

On Sep 21 2006, at 10:04, Ingo Molnar was caught saying:
>
> * Deepak Saxena <dsaxena.RemoveThis@plexity.net> wrote:
>
> > I am seeing an intermittent lock up on the ARM Versatile board during
> > the ALSA driver init that only shows up with (PREEMPT_RT &
> > !HIGH_RES_TIMERS & ARM_EABI) enabled. If HRT is disabled and EABI is
> > enabled, the kernel works every time, and same with !RT & !HRT & EABI.
> > I get no oops, just a complete lock up with no console output.
>
> does enabling LOCKDEP give you any better info? (It might not make a
> difference on the bootup that locks, but maybe you'll get a lockdep clue
> about the problem in one of the successful bootups.)

This is with LOCKDEP enabled. I'll look at this more tommorrow but
I think next steps for me are printks to see if I can pinpoint the
issue and possibly looking at the assembly to see if there's an obvious
compiler issue.

~Deepak

--
Deepak Saxena - dsaxena.RemoveThis@plexity.net - http://www.plexity.net

"An open heart has no possessions, only experiences" - Matt Bibbeau
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Esben Nielsen · External Since: Aug 14, 2006 Posts: 5

On Thu, 21 Sep 2006, Bill Huey wrote:

> On Thu, Sep 21, 2006 at 09:56:33AM +0200, Ingo Molnar wrote:
>> * Bill Huey <billh.TakeThisOut@gnuppy.monkey.org> wrote:
>>
>>> [...] If the upstream kernel used RCU function in a task allocation or
>>> task struct reading in the first place then call_rcu() would be a
>>> clear choice. However, I didn't see it used in that way (I could be
>>> wrong) [...]
>>
>> it was RCU-ified briefly but then it was further improved to direct
>> freeing, because upstream _can_ free it directly.
>
> Unfortunately, this is a problem with -rt patch and the lock ordering
> in this system when you have to call a memory allocator within an atomic
> critical section. I fully accept this as part of what goes into making a
> kernel preemptive and I'm ok with it. Not many folks know about the
> special case locking rules in the -rt kernel so this might be new to
> various folks.
>
> If you're looking for validation of this technique from me and an ego
> stroking, then you have it from me. Smile

>
> Fortunately, it's in a non-critical place so this should *not* be too
> much of a problem, but I've already encountered oddies trying to
> allocate a pool of entities for populating a free list under an atomic
> critical section of some sort for some code I've been writing. This is
> a significant problem with kernel coding in -rt, but I can't say what
> the general solution is other than making the memory allocators
> non-preemptible by reverting the locks back to raw spinlocks, etc...
> using lock-break, who knows. I'm ok with the current scenario, but this
> could eventually be a larger problem.
>

The whole point is to defer those frees to a task. In -rt call_rcu() is
abused to do that in the case of put_task_struct(). But it is abuse since
call_rcu() is much more resourcefull than simply defering to a task.

Paul's idea behind de-RCU'ing put_task_struct() is mostly performance and
partly readability because the extra RCU protection isn't needed.

So the answer is:
Make a general softirq to which free's can be defered from atomic regions,
don't abuse call_rcu().

Esben

> bill
>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar · External Since: May 15, 2006 Posts: 2457

* Esben Nielsen <nielsen.esben DeleteThis @gogglemail.com> wrote:

> The whole point is to defer those frees to a task. In -rt call_rcu()
> is abused to do that in the case of put_task_struct(). But it is abuse
> since call_rcu() is much more resourcefull than simply defering to a
> task.

nah, it's not nearly as "resourceful" as having a separate thread for
each teardown type ... It is also the easiest to maintain way of tearing
down stuff, so i'm quite happy with it. If then i'll move the mm_struct
teardown to RCU too. (Later on we can do some 'fastpath RCU' thing
perhaps, to force an RCU batch of work through quiescent state)

but this is quite likely a side-issue that probably has no connection to
the crash Bill was seeing. Whether separate teardown thread or
softirq-rcu, it's a separate thread already.

Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar · External Since: May 15, 2006 Posts: 2457

* Daniel Walker <dwalker.TakeThisOut@mvista.com> wrote:

> On Wed, 2006-09-20 at 22:14 +0200, Ingo Molnar wrote:
> > > if (up->port.sysrq) {
> > > /* serial8250_handle_port() already took the lock */
> > > locked = 0;
>
>
> In this case it had interrupts off in the !PREEMPT_RT case, but your
> change leaves them on here.. _irqsave only runs in two of the three
> cases..

doh, right you are. I updated to the patch below.

Ingo

Index: linux/drivers/serial/8250.c
===================================================================
--- linux.orig/drivers/serial/8250.c
+++ linux/drivers/serial/8250.c
@@ -2252,14 +2252,10 @@ serial8250_console_write(struct console

touch_nmi_watchdog();

- local_irq_save(flags);
- if (up->port.sysrq) {
- /* serial8250_handle_port() already took the lock */
- locked = 0;
- } else if (oops_in_progress) {
- locked = spin_trylock(&up->port.lock);
- } else
- spin_lock(&up->port.lock);
+ if (up->port.sysrq || oops_in_progress)
+ locked = spin_trylock_irqsave(&up->port.lock, flags);
+ else
+ spin_lock_irqsave(&up->port.lock, flags);

/*
* First save the IER then disable the interrupts
@@ -2281,8 +2277,7 @@ serial8250_console_write(struct console
serial_out(up, UART_IER, ier);

if (locked)
- spin_unlock(&up->port.lock);
- local_irq_restore(flags);
+ spin_unlock_irqrestore(&up->port.lock, flags);
}

static int serial8250_console_setup(struct console *co, char *options)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Daniel Walker · External Since: May 14, 2006 Posts: 391

On Thu, 2006-09-21 at 21:02 +0200, Ingo Molnar wrote:

> + if (up->port.sysrq || oops_in_progress)
> + locked = spin_trylock_irqsave(&up->port.lock, flags);
> + else
> + spin_lock_irqsave(&up->port.lock, flags);

Nice!

Daniel

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

john stultz · External Since: May 16, 2006 Posts: 225

On Wed, 2006-09-20 at 15:50 -0500, K.R. Foley wrote:
> Thomas Gleixner wrote:
> > On Wed, 2006-09-20 at 15:33 -0500, K.R. Foley wrote:
> >> DOH! The log had two different boots in it. Sad

Let's try this again. By
> >> the way, you may notice from my screw up that this is pretty much the
> >> same oops that I got with 2.6.17-rt*. I have been getting this on all of
> >> my SMP systems since we went past 2.6.16.
> >
> > Which module is modprobed ?
> >
> > tglx
> >
> >
> >
> How can I tell which particular module is being loaded? The last thing I
> see on the console before the oops is that it is starting udev. I am
> including the rest of the boot log below in hopes that will help.
> Suggestions? Something else I can provide?
>
[snip]
>kjournald starting. Commit interval 5 seconds
> EXT3-fs: mounted filesystem with ordered data mode.
> BUG: unable to handle kernel paging request at virtual address f3010000
> printing eip:
> *pde = 00000000
> Oops: 0000 [#1]
> PREEMPT SMP
> Modules linked in:
> CPU: 1
> EIP: 0060:[<c0131e02>] Not tainted VLI
> EFLAGS: 00010283 (2.6.18-rt2 #4)
> EIP is at lookup_symbol+0x11/0x35
> eax: 00000001 ebx: e0830e08 ecx: c036ff60 edx: c036dd94
> esi: f3010000 edi: e0830e08 ebp: df657e74 esp: df657e68
> ds: 007b es: 007b ss: 0068 preempt: 00000001
> Process modprobe (pid: 1366, ti=df656000 task=dfc68e90 task.ti=df656000)
> Stack: e083c780 00000c00 e0830e08 df657e90 c0131e6f df657ea8 df657ea4
> e083c780
> 00000c00 e0830e08 df657eb8 c0132c21 00000001 00000012 e082d074
> 00000000
> df657ecc e083a434 00000c00 e082d074 df657edc c0133188 e083c780
> 00000000
> Call Trace:
> [<c01037a1>] show_stack_log_lvl+0x87/0x8f
> [<c010391b>] show_registers+0x12f/0x198
> [<c0103b0c>] die+0x114/0x1c6
> [<c0111196>] do_page_fault+0x3f2/0x4c8
> [<c0103481>] error_code+0x39/0x40
> [<c0131e6f>] __find_symbol+0x25/0x2a5
> [<c0132c21>] resolve_symbol+0x27/0x5f
> [<c0133188>] simplify_symbols+0x83/0xf3
> [<c0133e65>] load_module+0x720/0xbb8
> [<c013435f>] sys_init_module+0x3f/0x1b5
> [<c0102969>] sysenter_past_esp+0x56/0x79
> Code: eb 11 8b 75 f0 41 83 c2 28 0f b7 46 30 39 c1 72 c9 31 c0 5a 59 5b
> 5e 5f 5d c3 55 89 e5 57 56 53 89 c3 39 ca 73 22 8b 72 04 89 df <ac> ae
> 75 08 84 c0 75 f8 31 c0 eb 04 19 c0 0c 01 85 c0 75 04 89
> EIP: [<c0131e02>] lookup_symbol+0x11/0x35 SS:ESP 0068:df657e68
>

I'm seeing a similar issue. Although the log is a bit futzed. Maybe its
the sd_mod?

at virtual address 75010000le kernel paging requestproc filesystem

printing eip:

Creating /dev
c01372d5
Starting udev
L*pde = cccccccc
oading sd_mod.kostopped custom tracer.
module
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU: 3
EIP: 0060:[<c01372d5>] Not tainted VLI
EFLAGS: 00010293 (2.6.18-rt3johnsmp #1)
EIP is at lookup_symbol+0x15/0x37
eax: ffffffff ebx: f88209a0 ecx: c03916fc edx: c038eba0
esi: 75010000 edi: f881a405 ebp: c7d01ea4 esp: c7d01e64
ds: 007b es: 007b ss: 0068 preempt: 00000001
Process insmod (pid: 462, ti=c7d00000 task=f7d6c1f0 task.ti=c7d00000)
Stack: f8821e00 f881a405 c0137345 f881a405 c038c658 c03916fc f88209a0 f8821e00
f881a405 00000000 c0138145 f881a405 c7d01ea0 c7d01ea4 00000001 00000000
f881a405 f88209a0 00000500 0000003c c01386d2 f88193d0 0000000b f881a405
Call Trace:
[<c0137345>] __find_symbol+0x26/0x2e0
[<c0138145>] resolve_symbol+0x23/0x5f
[<c01386d2>] simplify_symbols+0x7e/0xf0
[<c013942d>] load_module+0x7c4/0xc14
[<c01398dd>] sys_init_module+0x3d/0x1ae
[<c01029d9>] sysenter_past_esp+0x56/0x79
Code: c8 eb 0e 0f b7 45 30 41 83 c2 28 39 c1 72 ca 31 c0 5b 5e 5f 5d c3 57 56 8b 4c 24 14 8b 54 24 10 39 ca 73 24 8b 72 04 8b 7c 24 0c <ac> ae 75 08 84 c0 75 f8 31 c0 eb 04 19 c0 0c 01 85 c0 75 04 89
EIP: [<c01372d5>] lookup_symbol+0x15/0x37 SS:ESP 0068:c7d01e64
ERROR: /bin/insmod exited abnormally! (pid 462)
Loading jbd.ko module

thanks
-john

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

K.R. Foley · External Since: May 26, 2006 Posts: 17

john stultz wrote:
> On Wed, 2006-09-20 at 15:50 -0500, K.R. Foley wrote:
>> Thomas Gleixner wrote:
>>> On Wed, 2006-09-20 at 15:33 -0500, K.R. Foley wrote:
>>>> DOH! The log had two different boots in it. Sad

Let's try this again. By
>>>> the way, you may notice from my screw up that this is pretty much the
>>>> same oops that I got with 2.6.17-rt*. I have been getting this on all of
>>>> my SMP systems since we went past 2.6.16.
>>> Which module is modprobed ?
>>>
>>> tglx
>>>
>>>
>>>
>> How can I tell which particular module is being loaded? The last thing I
>> see on the console before the oops is that it is starting udev. I am
>> including the rest of the boot log below in hopes that will help.
>> Suggestions? Something else I can provide?
>>
> [snip]
>> kjournald starting. Commit interval 5 seconds
>> EXT3-fs: mounted filesystem with ordered data mode.
>> BUG: unable to handle kernel paging request at virtual address f3010000
>> printing eip:
>> *pde = 00000000
>> Oops: 0000 [#1]
>> PREEMPT SMP
>> Modules linked in:
>> CPU: 1
>> EIP: 0060:[<c0131e02>] Not tainted VLI
>> EFLAGS: 00010283 (2.6.18-rt2 #4)
>> EIP is at lookup_symbol+0x11/0x35
>> eax: 00000001 ebx: e0830e08 ecx: c036ff60 edx: c036dd94
>> esi: f3010000 edi: e0830e08 ebp: df657e74 esp: df657e68
>> ds: 007b es: 007b ss: 0068 preempt: 00000001
>> Process modprobe (pid: 1366, ti=df656000 task=dfc68e90 task.ti=df656000)
>> Stack: e083c780 00000c00 e0830e08 df657e90 c0131e6f df657ea8 df657ea4
>> e083c780
>> 00000c00 e0830e08 df657eb8 c0132c21 00000001 00000012 e082d074
>> 00000000
>> df657ecc e083a434 00000c00 e082d074 df657edc c0133188 e083c780
>> 00000000
>> Call Trace:
>> [<c01037a1>] show_stack_log_lvl+0x87/0x8f
>> [<c010391b>] show_registers+0x12f/0x198
>> [<c0103b0c>] die+0x114/0x1c6
>> [<c0111196>] do_page_fault+0x3f2/0x4c8
>> [<c0103481>] error_code+0x39/0x40
>> [<c0131e6f>] __find_symbol+0x25/0x2a5
>> [<c0132c21>] resolve_symbol+0x27/0x5f
>> [<c0133188>] simplify_symbols+0x83/0xf3
>> [<c0133e65>] load_module+0x720/0xbb8
>> [<c013435f>] sys_init_module+0x3f/0x1b5
>> [<c0102969>] sysenter_past_esp+0x56/0x79
>> Code: eb 11 8b 75 f0 41 83 c2 28 0f b7 46 30 39 c1 72 c9 31 c0 5a 59 5b
>> 5e 5f 5d c3 55 89 e5 57 56 53 89 c3 39 ca 73 22 8b 72 04 89 df <ac> ae
>> 75 08 84 c0 75 f8 31 c0 eb 04 19 c0 0c 01 85 c0 75 04 89
>> EIP: [<c0131e02>] lookup_symbol+0x11/0x35 SS:ESP 0068:df657e68
>>
>
>
> I'm seeing a similar issue. Although the log is a bit futzed. Maybe its
> the sd_mod?
>
> at virtual address 75010000le kernel paging requestproc filesystem
>
> printing eip:
>
> Creating /dev
> c01372d5
> Starting udev
> L*pde = cccccccc
> oading sd_mod.kostopped custom tracer.
> module
> Oops: 0000 [#1]
> PREEMPT SMP
> Modules linked in:
> CPU: 3
> EIP: 0060:[<c01372d5>] Not tainted VLI
> EFLAGS: 00010293 (2.6.18-rt3johnsmp #1)
> EIP is at lookup_symbol+0x15/0x37
> eax: ffffffff ebx: f88209a0 ecx: c03916fc edx: c038eba0
> esi: 75010000 edi: f881a405 ebp: c7d01ea4 esp: c7d01e64
> ds: 007b es: 007b ss: 0068 preempt: 00000001
> Process insmod (pid: 462, ti=c7d00000 task=f7d6c1f0 task.ti=c7d00000)
> Stack: f8821e00 f881a405 c0137345 f881a405 c038c658 c03916fc f88209a0 f8821e00
> f881a405 00000000 c0138145 f881a405 c7d01ea0 c7d01ea4 00000001 00000000
> f881a405 f88209a0 00000500 0000003c c01386d2 f88193d0 0000000b f881a405
> Call Trace:
> [<c0137345>] __find_symbol+0x26/0x2e0
> [<c0138145>] resolve_symbol+0x23/0x5f
> [<c01386d2>] simplify_symbols+0x7e/0xf0
> [<c013942d>] load_module+0x7c4/0xc14
> [<c01398dd>] sys_init_module+0x3d/0x1ae
> [<c01029d9>] sysenter_past_esp+0x56/0x79
> Code: c8 eb 0e 0f b7 45 30 41 83 c2 28 39 c1 72 ca 31 c0 5b 5e 5f 5d c3 57 56 8b 4c 24 14 8b 54 24 10 39 ca 73 24 8b 72 04 8b 7c 24 0c <ac> ae 75 08 84 c0 75 f8 31 c0 eb 04 19 c0 0c 01 85 c0 75 04 89
> EIP: [<c01372d5>] lookup_symbol+0x15/0x37 SS:ESP 0068:c7d01e64
> ERROR: /bin/insmod exited abnormally! (pid 462)
> Loading jbd.ko module
>
>
> thanks
> -john
>

Yes. This looks very much like the problem I am having. However, on the
system that this log came from I have no SCSI.

--
kr
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Lennert Buytenhek · External Since: Oct 21, 2005 Posts: 92

On Thu, Sep 21, 2006 at 10:19:21PM -0400, john cooper wrote:

> >ok, i've uploaded -rt3:
>
> Attached is a patch which fixes a build problem
> for ARM pxa270, and general ARM boot issue when
> LATENCY_TRACE is configured.
>
> -john
>
> --
> john.cooper.DeleteThis@third-harmonic.com

> include/asm-arm/arch-pxa/timex.h | 2 ++
> kernel/latency_trace.c | 2 ++
> 2 files changed, 4 insertions(+)
> =================================================================
> --- ./kernel/latency_trace.c.ORG 2006-09-20 21:10:15.000000000 -0400
> +++ ./kernel/latency_trace.c 2006-09-21 21:28:49.000000000 -0400
> @@ -150,6 +150,8 @@ enum trace_flag_type
> */
> #if !defined(CONFIG_DEBUG_PAGEALLOC) && !defined(CONFIG_SMP) && !defined(CONFIG_ARM)
> # define MAX_TRACE (unsigned long)(8192*16-1)
> +#elif defined(CONFIG_ARM) /* 4MB kernel image size limitation */
> +# define MAX_TRACE (unsigned long)(128*2-1)

This patch (queued for Linus) lifts that 4MB limitation:

http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=3809/2

(I ran into the limit when enabling lockdep on ARM.)

cheers,
Lennert
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar · External Since: May 15, 2006 Posts: 2457

* john stultz <johnstul.RemoveThis@us.ibm.com> wrote:

> I'm seeing a similar issue. Although the log is a bit futzed. Maybe
> its the sd_mod?
>
> at virtual address 75010000le kernel paging requestproc filesystem

would be nice to figure out why it crashes - unfortunately i cannot
trigger it. Could it be some build tool incompatibility perhaps? Some
sizing issue (some module struct gets too large)?

Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar · External Since: May 15, 2006 Posts: 2457

* Lennert Buytenhek <buytenh RemoveThis @wantstofly.org> wrote:

> On Thu, Sep 21, 2006 at 10:19:21PM -0400, john cooper wrote:
>
> > >ok, i've uploaded -rt3:
> >
> > Attached is a patch which fixes a build problem
> > for ARM pxa270, and general ARM boot issue when
> > LATENCY_TRACE is configured.
>
> This patch (queued for Linus) lifts that 4MB limitation:
>
> http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=3809/2
>
> (I ran into the limit when enabling lockdep on ARM.)

ok, i've added this no-4M-limit patch to -rt. John, does that solve your
problem?

Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Lee Revell · External Since: Nov 13, 2004 Posts: 177

On Wed, 2006-09-20 at 12:38 -0700, Mark Knecht wrote:
> On 9/20/06, Ingo Molnar <mingo.TakeThisOut@elte.hu> wrote:
> > I'm pleased to announce the 2.6.18-rt1 tree, which can be downloaded
> > from the usual place:
> >
> > http://redhat.com/~mingo/realtime-preempt/
> >
>
> Hi Ingo,
> I gave 2.6.18-rt2 a quick try. It compiled fine but crashed on
> boot. I've no way to copy the screen. I can send along a digital photo
> if the following isn't enough info.
>
> QUESTION: Should I be able to run ati-drivers-8.28.8 with this kernel
> or would I have to wait for ATI to put out a 2.6.18 compatible driver?
> The current version does not emerge with 2.6.18-rt2.
>

I would not expect any proprietary ATI driver to work with an -rt
kernel, unless they specifically release a binary for -rt.

> Not sure how much that will help you. Been awhile since I've sent
> along crash reports. I'll have to ge a second Linux machine running to
> do the console boot capture thing if you need it.

Try -rt3. If the crash persists, you could take a picture of the screen
with a digital camera and post a link to it.

Lee

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Daniel Walker · External Since: May 14, 2006 Posts: 391

On Thu, 2006-09-21 at 21:02 +0200, Ingo Molnar wrote:
> * Daniel Walker <dwalker.RemoveThis@mvista.com> wrote:
>
> > On Wed, 2006-09-20 at 22:14 +0200, Ingo Molnar wrote:
> > > > if (up->port.sysrq) {
> > > > /* serial8250_handle_port() already took the lock */
> > > > locked = 0;
> >
> >
> > In this case it had interrupts off in the !PREEMPT_RT case, but your
> > change leaves them on here.. _irqsave only runs in two of the three
> > cases..
>
> doh, right you are. I updated to the patch below.
>
> Ingo

On closer inspection I still think this is wrong. (Although it looks
really nice..) find below speaking only in term of !PREEMPT_RT ,

> Index: linux/drivers/serial/8250.c
> ===================================================================
> --- linux.orig/drivers/serial/8250.c
> +++ linux/drivers/serial/8250.c
> @@ -2252,14 +2252,10 @@ serial8250_console_write(struct console
>
> touch_nmi_watchdog();
>
> - local_irq_save(flags);
> - if (up->port.sysrq) {
> - /* serial8250_handle_port() already took the lock */
> - locked = 0;

in the old version interrupts are off, and stay off until the function
returns in all cases. Even "locked = 0" .

> - } else if (oops_in_progress) {
> - locked = spin_trylock(&up->port.lock);
> - } else
> - spin_lock(&up->port.lock);
> + if (up->port.sysrq || oops_in_progress)
> + locked = spin_trylock_irqsave(&up->port.lock, flags);

Now in the new version interrupts are only off if you _get the lock_.
Presumably the lock is taken in the calling function, but interrupts
aren't disabled.

I'm assuming the code is disabling interrupts for a good reason, I don't
know enough about the code to say it isn't.

> + else
> + spin_lock_irqsave(&up->port.lock, flags);
>
> /*
> * First save the IER then disable the interrupts
> @@ -2281,8 +2277,7 @@ serial8250_console_write(struct console
> serial_out(up, UART_IER, ier);
>
> if (locked)
> - spin_unlock(&up->port.lock);
> - local_irq_restore(flags);
> + spin_unlock_irqrestore(&up->port.lock, flags);
> }
>
> static int serial8250_console_setup(struct console *co, char *options)
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo.RemoveThis@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Mike Kravetz · External Since: May 16, 2006 Posts: 10

On Wed, Sep 20, 2006 at 04:19:07PM +0200, Ingo Molnar wrote:
> In particular, a nasty softirq performance bug has been fixed, which
> caused the "5x slowdown under TCP" bug reported to lkml - those TCP
> performance figures are now on par with vanilla performance.

Just curious about the cause and fix for this issue. I tried searching
the mail lists for discussion but came up empty. The patch it too big
for me to determine what specific changes addressed this issue. If anyone
can point me in the right direction, that would be appreciated.

Thanks,
--
Mike
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Florian Schmidt · External Since: Sep 26, 2006 Posts: 2

On Monday 25 September 2006 11:53, Florian Schmidt wrote:
> .config attached
>
> Besides these problems it ran very well for the hours i had it up. Nothing
> suspicious in the logs. Will try another build with more aggressive
> debugging options.

Erm, forget this bug report.

- The long switching to console time seems to be duee to the xorg nv driver

- logging into X via gdm now fails under different kernels, too (though i
still have no clue why)

- also in 2.6.17-rt8 i can run jack with rt even though no rt-lsm is loaded.
This is the only really funky one, but i suspect there's some other mechanism
that explains this.

Regarsd,
Flo

--
Palimm Palimm!
http://tapas.affenbande.org
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Eric W. Biederman · External Since: May 19, 2006 Posts: 1085

Bill Huey (hui) <billh.RemoveThis@gnuppy.monkey.org> writes:

> On Wed, Sep 20, 2006 at 04:19:07PM +0200, Ingo Molnar wrote:
>> I'm pleased to announce the 2.6.18-rt1 tree, which can be downloaded
>> from the usual place:
> ...
>> as usual, bugreports, fixes and suggestions are welcome,
>
> Speaking of which...
>
> This patch moves put_task_struct() reaping into a thread instead of an
> RCU callback function as discussed with Esben publically and Ingo privately:

Stupid question.

Why does the rt tree make all calls to put_task_struct an rcu action?
We only need the rcu callback from kernel/exit.c

Nothing else needs those semantics.

I agree that put_task_struct is the most common point so this is unlikely
to remove your issues with rcu callbacks but it just seems completely backwards
to increase the number of rcu callbacks in the rt tree.

Eric
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Bill Huey · External Since: Jul 24, 2006 Posts: 84

On Tue, Sep 26, 2006 at 08:55:41PM -0600, Eric W. Biederman wrote:
> Bill Huey (hui) <billh RemoveThis @gnuppy.monkey.org> writes:
> > This patch moves put_task_struct() reaping into a thread instead of an
> > RCU callback function as discussed with Esben publically and Ingo privately:
>
> Stupid question.

It's a great question actually.

> Why does the rt tree make all calls to put_task_struct an rcu action?
> We only need the rcu callback from kernel/exit.c

Because the conversion of memory allocation routines like kmalloc and kfree aren't
safely callable within a preempt_disable critical section since they were incompletely
converted in the -rt. It can run into the sleeping in atomic scenario which can result
in a deadlock since those routines use blocking locks internally in the implementation
now as a result of the spinlock_t conversion to blocking locks.

> Nothing else needs those semantics.

Right, blame it on the incomplete conversion of the kmalloc and friends. GFP_ATOMIC is
is kind of meaningless in the -rt tree and it might be a good thing to add something
like GFP_RT_ATOMIC for cases like this to be handled properly and restore that particular
semantic in a more meaningful way.

> I agree that put_task_struct is the most common point so this is unlikely
> to remove your issues with rcu callbacks but it just seems completely backwards
> to increase the number of rcu callbacks in the rt tree.

I'm not sure what mean here, but if you mean that you don't like the RCU API abuse then
I agree with you on that. However, Ingo disagrees and I'm not going to argue it with him.
Although, I'm not going stop you if you do. Smile

bill

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Eric W. Biederman · External Since: May 19, 2006 Posts: 1085

Bill Huey (hui) <billh RemoveThis @gnuppy.monkey.org> writes:

> On Tue, Sep 26, 2006 at 08:55:41PM -0600, Eric W. Biederman wrote:
>> Bill Huey (hui) <billh RemoveThis @gnuppy.monkey.org> writes:
>> > This patch moves put_task_struct() reaping into a thread instead of an
>> > RCU callback function as discussed with Esben publically and Ingo privately:
>>
>> Stupid question.
>
> It's a great question actually.
>
>> Why does the rt tree make all calls to put_task_struct an rcu action?
>> We only need the rcu callback from kernel/exit.c
>
> Because the conversion of memory allocation routines like kmalloc and kfree aren't
> safely callable within a preempt_disable critical section since they were incompletely
> converted in the -rt. It can run into the sleeping in atomic scenario which can result
> in a deadlock since those routines use blocking locks internally in the implementation
> now as a result of the spinlock_t conversion to blocking locks.

Interesting. I think the easy solution would just be to assert that put_task_struct
can sleep and to fix any callers that expect differently. I haven't looked very
closely but I don't recall anything that needs put_task_struct to be atomic.
With a function that complex I certainly would not expect it to never sleep unless
it had a big fat comment.

Well I did find an instance where we call put_task_struct with a
spinlock held. Inside of lib/rwsem.c:rwsem_down_failed_common().

Still that may be the only user that cares. I suspect with a little
code rearrangement that case is fixable. It's not like that code is a
fast path or anything. It should just be a matter of passing the
task struct outside of the spinlock before calling put_task_struct.

>> Nothing else needs those semantics.
>
> Right, blame it on the incomplete conversion of the kmalloc and friends. GFP_ATOMIC is
> is kind of meaningless in the -rt tree and it might be a good thing to add something
> like GFP_RT_ATOMIC for cases like this to be handled properly and restore that
> particular semantic in a more meaningful way.

But this is a path where we are freeing data, so GFP_ATOMIC should not come
into it. I just read through the code and there are not allocations
there.

>> I agree that put_task_struct is the most common point so this is unlikely
>> to remove your issues with rcu callbacks but it just seems completely backwards
>> to increase the number of rcu callbacks in the rt tree.
>
> I'm not sure what mean here, but if you mean that you don't like the RCU API abuse then
> I agree with you on that. However, Ingo disagrees and I'm not going to argue it with him.
> Although, I'm not going stop you if you do. Smile

What I was thinking is that rcu isn't terribly friendly to realtime
activities because it postpones work and can wind up with a lot of
work to do at some random time later which can be bad for latencies.

So I was very surprised to see an rt patch making more things under
rcu protection. Especially as I have heard other developers worried
about rt issues discussing removing the rcu functionality.

My gut feel now that I understand the pieces is that this approach has
all of the hallmarks of a hack, both the kmalloc/kfree thing and even
more calling put_task_struct in an atomic context. If the callers
were fixed put_task_struct could safely sleep so kmalloc/kfree
sleeping would not be a problem.

Eric

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Bill Huey · External Since: Jul 24, 2006 Posts: 84

On Wed, Sep 27, 2006 at 12:02:21AM -0600, Eric W. Biederman wrote:
> Bill Huey (hui) <billh.RemoveThis@gnuppy.monkey.org> writes:
> > Because the conversion of memory allocation routines like kmalloc and kfree aren't
> > safely callable within a preempt_disable critical section since they were incompletely
> > converted in the -rt. It can run into the sleeping in atomic scenario which can result
> > in a deadlock since those routines use blocking locks internally in the implementation
> > now as a result of the spinlock_t conversion to blocking locks.
>
> Interesting. I think the easy solution would just be to assert that put_task_struct
> can sleep and to fix any callers that expect differently. I haven't looked very
> closely but I don't recall anything that needs put_task_struct to be atomic.
> With a function that complex I certainly would not expect it to never sleep unless
> it had a big fat comment.

One of the main claims about the -rt patch is that the kernel is basically
conversion of the current locking semantics in the kernel. What you mentioned
above would deviate from that and and clutter non-preemptive kernel maintenance.
If you're suggesting a more general change to that function, then it wouldn't
violate that.

> Well I did find an instance where we call put_task_struct with a
> spinlock held. Inside of lib/rwsem.c:rwsem_down_failed_common().
>
> Still that may be the only user that cares. I suspect with a little
> code rearrangement that case is fixable. It's not like that code is a
> fast path or anything. It should just be a matter of passing the
> task struct outside of the spinlock before calling put_task_struct.

Yeah, that's what the small patch of mine does outside of doing an RCU
callback.

> > Right, blame it on the incomplete conversion of the kmalloc and friends. GFP_ATOMIC is
> > is kind of meaningless in the -rt tree and it might be a good thing to add something
> > like GFP_RT_ATOMIC for cases like this to be handled properly and restore that
> > particular semantic in a more meaningful way.
>
> But this is a path where we are freeing data, so GFP_ATOMIC should not come
> into it. I just read through the code and there are not allocations
> there.

Correct, I see it as a possibly bigger problem since memory allocators and
destructors aren't available readily in preempt_disable critical sections.
It's not the case in the regular kernel which could be a significant concern.

> > I'm not sure what mean here, but if you mean that you don't like the RCU API abuse then
> > I agree with you on that. However, Ingo disagrees and I'm not going to argue it with him.
> > Although, I'm not going stop you if you do. Smile

>
> What I was thinking is that rcu isn't terribly friendly to realtime
> activities because it postpones work and can wind up with a lot of
> work to do at some random time later which can be bad for latencies.
>
> So I was very surprised to see an rt patch making more things under
> rcu protection. Especially as I have heard other developers worried
> about rt issues discussing removing the rcu functionality.

There are a couple of issues here.

(1) The RCU callback isn't used in this case to back other RCU read critical
sections. It's just used as a generic callback mechanism in this case. Some
consider it a hack.

(2) RCU isn't necessarily bad for -rt since Paul McKenney and folks are
working on making it preempt friendly. Any talk of removing the use RCU in -rt
is premature and probably unlikely because of this work. There are many options
here. RCU very useful for scalability so seeing it go away in -rt would be
generally a bad thing IMO.

The use of lock-free techinques is something that could eventually be more
widely used in -rt since it'll make kernel call paths potentially more
deterministic without worrying about hitting a contending mutex and that can
effect determinism. It's not expect to be the case where calls into the kernel
are going to be deterministic but to extend the basic idea of determinism into
certain critical kernel paths like a read() to a special device driver, etc...

Route table look up also comes to mind here that benefits from this case if
there's a need for real time facilities in networking.

Just some ideas here.

> My gut feel now that I understand the pieces is that this approach has
> all of the hallmarks of a hack, both the kmalloc/kfree thing and even
> more calling put_task_struct in an atomic context. If the callers
> were fixed put_task_struct could safely sleep so kmalloc/kfree
> sleeping would not be a problem.

I can't say. The current logic just queues the request into a list and a
thread wakes to reap that task_struct. It's sufficient and the need to
convert put_task_struct so that it can block.

bill

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/