2.6.18-rt1

Lee Revell · External Since: Nov 13, 2004 Posts: 177

On Wed, 2006-09-20 at 16:19 +0200, Ingo Molnar wrote:
> I'm pleased to announce the 2.6.18-rt1 tree, which can be downloaded
> from the usual place:
>
> http://redhat.com/~mingo/realtime-preempt/

I got this Oops with -rt3, looks RCU related. Apologies in advance if
it's already known.

Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
[<ffffffff802aafa7>] __rcu_read_unlock+0x2e/0x82
PGD 46a3067 PUD 4e27067 PMD 0
Oops: 0002 [1] PREEMPT SMP
CPU 1
Modules linked in: rfcomm hidp l2cap bluetooth nfsd exportfs lockd nfs_acl sunrpc powernow_k8 cpufreq_userspace cpufreq_stats freq_table cpufreq_powersave cpufreq_ondemand cpufreq_conservative video button battery container ac asus_acpi dm_mod md_mod sr_mod sbp2 lp psmouse serio_raw sg evdev floppy parport_pc parport pcspkr i2c_nforce2 i2c_core shpchp pci_hotplug ext3 jbd mbcache ohci_hcd ehci_hcd ohci1394 ieee1394 forcedeth ide_generic ide_cd cdrom sd_mod sata_nv libata scsi_mod generic amd74xx ide_core thermal processor fan
Pid: 21372, comm: fixdep Not tainted 2.6.18-rt3-smp-noipv6 #1
RIP: 0010:[<ffffffff802aafa7>] [<ffffffff802aafa7>] __rcu_read_unlock+0x2e/0x82
RSP: 0018:ffff8100045dbbf0 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff810015a63070 RCX: 0000000000000246
RDX: 0000000000000000 RSI: ffff8100188bf007 RDI: ffff810015a63078
RBP: ffff810015a63070 R08: 0000000000000000 R09: ffff81001212d6c0
R10: ffff81001b9c0d48 R11: ffffffff8022ae94 R12: ffff810015a63078
R13: ffff8100119da4b0 R14: ffff8100045dbca8 R15: 0000000071bf92cd
FS: 00002b7fb742d6d0(0000) GS:ffff81001b9178c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 00000000194c3000 CR4: 00000000000006e0
Process fixdep (pid: 21372, threadinfo ffff8100045da000, task ffff81001622d800)
Stack: ffffffff80208db8 0000000100000105 0000000700000001 ffff8100188bf000
00000000000041ed ffff8100045dbea8 ffff8100119f5180 ffff810001232480
ffff8100045dbea8 ffff8100045dbca8 ffffffff8020c353 ffff81001b9c0d00
Call Trace:
[<ffffffff80208db8>] __d_lookup+0x10b/0x11d
[<ffffffff8020c353>] do_lookup+0x2a/0x173
[<ffffffff80209179>] __link_path_walk+0x3af/0xf5c
[<ffffffff8020c537>] file_read_actor+0x0/0xce
[<ffffffff80299575>] _atomic_dec_and_spin_lock+0x2b/0x33
[<ffffffff8020dc2c>] link_path_walk+0x5c/0xe5
[<ffffffff8029955e>] _atomic_dec_and_spin_lock+0x14/0x33
[<ffffffff80213fe4>] get_unused_fd+0xf9/0x107
[<ffffffff802117be>] filemap_nopage+0x1cb/0x39b
[<ffffffff8020c201>] do_path_lookup+0x26c/0x290
[<ffffffff8029a09b>] __rt_rwlock_init+0x9/0x12
[<ffffffff80221a87>] __path_lookup_intent_open+0x56/0x97
[<ffffffff8021909c>] open_namei+0x6d/0x6a4
[<ffffffff8021ca81>] free_pgtables+0xe6/0x172
[<ffffffff802bbada>] __cache_free+0x51/0x1f2
[<ffffffff80225776>] do_filp_open+0x1c/0x3d
[<ffffffff8029955e>] _atomic_dec_and_spin_lock+0x14/0x33
[<ffffffff80213fe4>] get_unused_fd+0xf9/0x107
[<ffffffff80217e96>] do_sys_open+0x44/0xc5
[<ffffffff8025b1ce>] system_call+0x7e/0x83

Code: f0 ff 08 65 48 8b 04 25 00 00 00 00 48 c7 80 a8 00 00 00 00
RIP [<ffffffff802aafa7>] __rcu_read_unlock+0x2e/0x82
RSP <ffff8100045dbbf0>
CR2: 0000000000000000

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Dipankar Sarma · External Since: Jun 27, 2006 Posts: 61

On Sat, Sep 30, 2006 at 02:06:04PM -0400, Lee Revell wrote:
> On Wed, 2006-09-20 at 16:19 +0200, Ingo Molnar wrote:
> > I'm pleased to announce the 2.6.18-rt1 tree, which can be downloaded
> > from the usual place:
> >
> > http://redhat.com/~mingo/realtime-preempt/
>
> I got this Oops with -rt3, looks RCU related. Apologies in advance if
> it's already known.
>
> Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
> [<ffffffff802aafa7>] __rcu_read_unlock+0x2e/0x82
> PGD 46a3067 PUD 4e27067 PMD 0
> Oops: 0002 [1] PREEMPT SMP
> CPU 1

I see a very similar crash while running rcutorture on 2.6.18-mm1 and
my rcu patchset that has rcupreempt stuff rom -rt. I don't see this
while running on 2.6.18-rc3, but then rc3 had an older version
of rcutorture. I am working on narrowing it down.

The following script reproduces the problem quickly (within
a couple of minutes) in my 4-cpu x86_64 system -

#! /bin/sh
for ((i=0 ; i<200 ; i++))
do
echo "Starting pass $i"
modprobe rcutorture stat_interval=10 # test_no_idle_hz=1 shuffle_interval=5
sleep 30
rmmod rcutorture
dmesg | sed -n -e '/^rcutorture: --- End of test:/p' | tail -1
done
exit 0

Thanks
Dipankar
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Lee Revell · External Since: Nov 13, 2004 Posts: 177

On Sat, 2006-09-30 at 23:48 +0530, Dipankar Sarma wrote:
> On Sat, Sep 30, 2006 at 02:06:04PM -0400, Lee Revell wrote:
> > I got this Oops with -rt3, looks RCU related. Apologies in advance if
> > it's already known.
> >
> > Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
> > [<ffffffff802aafa7>] __rcu_read_unlock+0x2e/0x82
> > PGD 46a3067 PUD 4e27067 PMD 0
> > Oops: 0002 [1] PREEMPT SMP
> > CPU 1
>
> I see a very similar crash while running rcutorture on 2.6.18-mm1 and
> my rcu patchset that has rcupreempt stuff rom -rt. I don't see this
> while running on 2.6.18-rc3, but then rc3 had an older version
> of rcutorture. I am working on narrowing it down.
>
> The following script reproduces the problem quickly (within
> a couple of minutes) in my 4-cpu x86_64 system -

Let me know if you want more info such as config. I was compiling a
kernel when the Oops occurred. System is Athlon X2.

Lee

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Karsten Wiese · External Since: Jun 21, 2006 Posts: 14

Am Samstag, 30. September 2006 20:18 schrieb Dipankar Sarma:
> On Sat, Sep 30, 2006 at 02:06:04PM -0400, Lee Revell wrote:
> > On Wed, 2006-09-20 at 16:19 +0200, Ingo Molnar wrote:
> > > I'm pleased to announce the 2.6.18-rt1 tree, which can be downloaded
> > > from the usual place:
> > >
> > > http://redhat.com/~mingo/realtime-preempt/
> >
> > I got this Oops with -rt3, looks RCU related. Apologies in advance if
> > it's already known.
> >
> > Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
> > [<ffffffff802aafa7>] __rcu_read_unlock+0x2e/0x82
> > PGD 46a3067 PUD 4e27067 PMD 0
> > Oops: 0002 [1] PREEMPT SMP
> > CPU 1
>
> I see a very similar crash while running rcutorture on 2.6.18-mm1 and
> my rcu patchset that has rcupreempt stuff rom -rt. I don't see this
> while running on 2.6.18-rc3, but then rc3 had an older version
> of rcutorture. I am working on narrowing it down.
>
> The following script reproduces the problem quickly (within
> a couple of minutes) in my 4-cpu x86_64 system -
>
> #! /bin/sh
> for ((i=0 ; i<200 ; i++))
> do
> echo "Starting pass $i"
> modprobe rcutorture stat_interval=10 # test_no_idle_hz=1 shuffle_interval=5
> sleep 30
> rmmod rcutorture
> dmesg | sed -n -e '/^rcutorture: --- End of test:/p' | tail -1
> done
> exit 0
>

Bug just happened here on a tainted UP x86_64 running rt4.
IIRC this is the second time in 2 weeks or so.
Machine seams to be fine still after the oops...

<Oops>
Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
[<ffffffff802a1b21>] __rcu_read_unlock+0x2e/0x80
PGD 3b616067 PUD 1718b067 PMD 0
Oops: 0002 [1] PREEMPT
CPU 0
Modules linked in: autofs4 sunrpc video button ac lp parport_pc parport nvram snd_via82xx gameport snd_ac97_codec snd_ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq nvidia snd_pcm_oss snd_mixer_oss snd_pcm ehci_hcd uhci_hcd snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi pcspkr snd_seq_device snd i2c_viapro i2c_core r8169 soundcore ext3 jbd
Pid: 7102, comm: sh Tainted: P 2.6.18-rt4 #4
RIP: 0010:[<ffffffff802a1b21>] [<ffffffff802a1b21>] __rcu_read_unlock+0x2e/0x80
RSP: 0018:ffff8100189ebc00 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff81003fd0d450 RCX: 0000000000000246
RDX: 0000000000000000 RSI: ffff81003e351008 RDI: ffff81003fd0d458
RBP: ffff81003fd0d450 R08: ffff81003e351005 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff8022a714 R12: ffff81003fd0d458
R13: ffff81003f43b070 R14: ffff8100189ebcb8 R15: 000000000023605a
FS: 00002ac8f7d9dd50(0000) GS:ffffffff8053f000(0000) knlGS:00000000f7fa96c0
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000003582f000 CR4: 00000000000006e0
Process sh (pid: 7102, threadinfo ffff8100189ea000, task ffff81000c8d2080)
Stack: ffffffff80208d3b 00000001000200d2 00000003804686b0 ffff81003e351005
00000000000041ed ffff8100189ebe48 ffff81003f7bd788 ffff81003fde8cc0
ffff8100189ebe48 ffff8100189ebcb8 ffffffff8020be2c 00000000000041ed
Call Trace:
[<ffffffff80208d3b>] __d_lookup+0x10a/0x11c
[<ffffffff8020be2c>] do_lookup+0x2a/0x173
[<ffffffff802090f9>] __link_path_walk+0x3ac/0xf4a
[<ffffffff8020d93b>] link_path_walk+0x5a/0xe1
[<ffffffff8020bc82>] do_path_lookup+0x26d/0x2e9
[<ffffffff80210f67>] getname+0x15b/0x1c1
[<ffffffff802216e3>] __user_walk_fd+0x37/0x4c
[<ffffffff802265ce>] vfs_stat_fd+0x1b/0x4a
[<ffffffff8022143c>] sys_newstat+0x19/0x31
[<ffffffff8025a7a1>] error_exit+0x0/0x84
[<ffffffff80259ece>] system_call+0x7e/0x83

Code: ff 08 65 48 8b 04 25 00 00 00 00 48 c7 80 a8 00 00 00 00 00
RIP [<ffffffff802a1b21>] __rcu_read_unlock+0x2e/0x80
RSP <ffff8100189ebc00>
CR2: 0000000000000000
</Oops>

Thanks

Karsten
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Dipankar Sarma · External Since: Jun 27, 2006 Posts: 61

On Fri, Oct 13, 2006 at 11:18:01PM +0200, Karsten Wiese wrote:
> Am Samstag, 30. September 2006 20:18 schrieb Dipankar Sarma:
> > On Sat, Sep 30, 2006 at 02:06:04PM -0400, Lee Revell wrote:
> > > On Wed, 2006-09-20 at 16:19 +0200, Ingo Molnar wrote:
> > > > I'm pleased to announce the 2.6.18-rt1 tree, which can be downloaded
> > > > from the usual place:
> > > >
> > > > http://redhat.com/~mingo/realtime-preempt/
> > >
> > > I got this Oops with -rt3, looks RCU related. Apologies in advance if
> > > it's already known.
> > >
> > > Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
> > > [<ffffffff802aafa7>] __rcu_read_unlock+0x2e/0x82
> > > PGD 46a3067 PUD 4e27067 PMD 0
> > > Oops: 0002 [1] PREEMPT SMP
> > > CPU 1
> >
> > I see a very similar crash while running rcutorture on 2.6.18-mm1 and
> > my rcu patchset that has rcupreempt stuff rom -rt. I don't see this
> >
>
> Bug just happened here on a tainted UP x86_64 running rt4.
> IIRC this is the second time in 2 weeks or so.
> Machine seams to be fine still after the oops...
>
> <Oops>
> Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
> [<ffffffff802a1b21>] __rcu_read_unlock+0x2e/0x80
> PGD 3b616067 PUD 1718b067 PMD 0
> Oops: 0002 [1] PREEMPT
> CPU 0
> Modules linked in: autofs4 sunrpc video button ac lp parport_pc parport nvram snd_via82xx gameport snd_ac97_codec snd_ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq nvidia snd_pcm_oss snd_mixer_oss snd_pcm ehci_hcd uhci_hcd snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi pcspkr snd_seq_device snd i2c_viapro i2c_core r8169 soundcore ext3 jbd
> Pid: 7102, comm: sh Tainted: P 2.6.18-rt4 #4
> RIP: 0010:[<ffffffff802a1b21>] [<ffffffff802a1b21>] __rcu_read_unlock+0x2e/0x80

Sorry, I should have published my investigations long ago. I tracked
this down (atleast the crash in my machine) to NMI interference
with rcu_read_lock()/rcu_read_unlock(). We use those APIs
from NMI context as well
(default_do_nmi()->notify_die()->atomic_notifier_call_chain()).

Can you try with nmi_watchdog=0 in the kernel command line ?

Paul has an NMI-safe patch for rcupreempt which I am adopting
and testing at the moment. If this works well, I will publish
a new patchset.

Thanks
Dipankar
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Lee Revell · External Since: Nov 13, 2004 Posts: 177

On Fri, 2006-10-13 at 23:18 +0200, Karsten Wiese wrote:
> Bug just happened here on a tainted UP x86_64 running rt4.
> IIRC this is the second time in 2 weeks or so.
> Machine seams to be fine still after the oops...

I just hit it again with (untainted) 2.6.18-rt5.

Lee

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Lee Revell · External Since: Nov 13, 2004 Posts: 177

On Sat, 2006-10-14 at 02:54 +0530, Dipankar Sarma wrote:
> Can you try with nmi_watchdog=0 in the kernel command line ?
>
> Paul has an NMI-safe patch for rcupreempt which I am adopting
> and testing at the moment. If this works well, I will publish
> a new patchset.
>

The bug is too hard to hit for me to provide useful feedback. I've only
seen it once since my original report.

FWIW, I am also seeing hard lockups every 12-24 hours but the box is
headless and I don't have the bandwidth to debug these further. It was
stable with 2.6.17-rt*.

Lee

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Dipankar Sarma · External Since: Jun 27, 2006 Posts: 61

On Fri, Oct 13, 2006 at 06:12:16PM -0400, Lee Revell wrote:
> On Sat, 2006-10-14 at 02:54 +0530, Dipankar Sarma wrote:
> > Can you try with nmi_watchdog=0 in the kernel command line ?
> >
> > Paul has an NMI-safe patch for rcupreempt which I am adopting
> > and testing at the moment. If this works well, I will publish
> > a new patchset.
> >
>
> The bug is too hard to hit for me to provide useful feedback. I've only
> seen it once since my original report.
>
> FWIW, I am also seeing hard lockups every 12-24 hours but the box is
> headless and I don't have the bandwidth to debug these further. It was
> stable with 2.6.17-rt*.

Can you try whatever you were doing with nmi_watchdog=0 ? If it is
stable, then that would explain the problem. I believe Andi enabled
nmi watchdog on x86_64 by default recently, that might be why
we are seeing it now.

Thanks
Dipankar
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Lee Revell · External Since: Nov 13, 2004 Posts: 177

On Sat, 2006-10-14 at 03:46 +0530, Dipankar Sarma wrote:
> > FWIW, I am also seeing hard lockups every 12-24 hours but the box is
> > headless and I don't have the bandwidth to debug these further. It
> was
> > stable with 2.6.17-rt*.
>
> Can you try whatever you were doing with nmi_watchdog=0 ? If it is
> stable, then that would explain the problem. I believe Andi enabled
> nmi watchdog on x86_64 by default recently, that might be why
> we are seeing it now.

Looks like that was the problem, the hard lockups are gone.

Lee

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar · External Since: May 15, 2006 Posts: 3599

* Dipankar Sarma wrote:

> Sorry, I should have published my investigations long ago. I tracked
> this down (atleast the crash in my machine) to NMI interference with
> rcu_read_lock()/rcu_read_unlock(). We use those APIs from NMI context
> as well
> (default_do_nmi()->notify_die()->atomic_notifier_call_chain()).
>
> Can you try with nmi_watchdog=0 in the kernel command line ?
>
> Paul has an NMI-safe patch for rcupreempt which I am adopting and
> testing at the moment. If this works well, I will publish a new
> patchset.

spent some good time debugging this 2 weeks ago and added the fix below
to rt5, but i forgot to do the symmetric fix for x86_64...

Ingo

----------->
arch/i386/kernel/traps.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

Index: linux/arch/i386/kernel/traps.c
===================================================================
--- linux.orig/arch/i386/kernel/traps.c
+++ linux/arch/i386/kernel/traps.c
@@ -716,9 +716,6 @@ static void default_do_nmi(struct pt_reg
reason = get_nmi_reason();

if (!(reason & 0xc0)) {
- if (notify_die(DIE_NMI_IPI, "nmi_ipi", regs, reason, 2, SIGINT)
- == NOTIFY_STOP)
- return;
#ifdef CONFIG_X86_LOCAL_APIC
/*
* Ok, so this is none of the documented NMI sources,
@@ -729,6 +726,9 @@ static void default_do_nmi(struct pt_reg
return;
}
#endif
+ if (notify_die(DIE_NMI_IPI, "nmi_ipi", regs, reason, 2, SIGINT)
+ == NOTIFY_STOP)
+ return;
unknown_nmi_error(reason, regs);
return;
}

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar · External Since: May 15, 2006 Posts: 3599

* Lee Revell wrote:

> > Can you try whatever you were doing with nmi_watchdog=0 ? If it is
> > stable, then that would explain the problem. I believe Andi enabled
> > nmi watchdog on x86_64 by default recently, that might be why we are
> > seeing it now.
>
> Looks like that was the problem, the hard lockups are gone.

ok. Meanwhile i discovered that i fixed this bug on i686 but not on
x86_64. Could you try -rt6, does it work with the NMI watchdog
re-enabled?

Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/