Kernel debug howto

George · External Since: Jul 16, 2004 Posts: 28

Hi all,

I'm developing a kernel module for networking purposes.

I've been reading for debugging tools and some time ago I used kgdb, but it
is not available for kernel 2.6.20.

My module is making a kernel panic and I cannot see the point where it
crash. Is there any way to get the ouput? Any way of debugging this kernel
version?

Using Ubuntu 7.04

TA

Rainer Weikusat · External Since: Apr 02, 2007 Posts: 100

George <george.news.DeleteThis@NOSPAMgmx.net> writes:
> I'm developing a kernel module for networking purposes.
>
> I've been reading for debugging tools and some time ago I used kgdb, but it
> is not available for kernel 2.6.20.
>
> My module is making a kernel panic and I cannot see the point where it
> crash.

You should be getting an 'Oops' printed on the console and you can use
the ksymoops program to decode that.

> Is there any way to get the ouput?

It usually possible to locate a problem like this by liberal use of
'printk'.

George · External Since: Jul 16, 2004 Posts: 28

Hi,

Rainer Weikusat <rweikusat.TakeThisOut@mssgmbh.com> wrote in
news:87abuvvy75.fsf@fever.mssgmbh.com:

> George <george.news.TakeThisOut@NOSPAMgmx.net> writes:
>> I'm developing a kernel module for networking purposes.
>>
>> I've been reading for debugging tools and some time ago I used kgdb,
>> but it is not available for kernel 2.6.20.
>>
>> My module is making a kernel panic and I cannot see the point where
>> it crash.
>
> You should be getting an 'Oops' printed on the console and you can use
> the ksymoops program to decode that.

This is what I'm doing now. The problem is that the Oops is printed and
halt the computer. Any tip on how to store this opps?

>> Is there any way to get the ouput?
>
> It usually possible to locate a problem like this by liberal use of
> 'printk'.

Again, what I'm doing. The problem is that the opps clears the screen and
doesn't let me see where it crash Sad

And as the computer hangs it is
imposible to scroll.

Help!!! Wink

--
Saludos a todos
#
# Contra el terrorismo
#

Tim Southerwood · External Since: Apr 23, 2007 Posts: 113

George wrote:

>
> This is what I'm doing now. The problem is that the Oops is printed and
> halt the computer. Any tip on how to store this opps?
>

The easiest way is to wire a spare serial port to another PC via aNull
Modem cable, run minicom or equivalent on the other PC and boot your
troubled machine with an extra pair of boot parameters:

console=ttyS0,38400 console=tty0

This assumes COM1, change ttyS0 to ttyS1 for COM2

Obviously change 38400 baud to whatever works best for you and set the other
end for the same baud, no parity, 8 bits, XON/XOFF flow control only,
disable DTR and all other hardware line controls.

Set up minicom or equiv for logging to a file, or have a decent scroll
buffer set in the terminal and you're good to go. Linux will generally try
to get messages out over serial even if everything else is dead.

HTH

Tim

George · External Since: Jul 16, 2004 Posts: 28

Hi,

Tim Southerwood <ts RemoveThis @dionic.net> wrote in
news:4678f1af$0$645$5a6aecb4@news.aaisp.net.uk:

> George wrote:
>
>>
>> This is what I'm doing now. The problem is that the Oops is printed
>> and halt the computer. Any tip on how to store this opps?
>>
>
> The easiest way is to wire a spare serial port to another PC via
> aNull Modem cable, run minicom or equivalent on the other PC and boot
> your troubled machine with an extra pair of boot parameters:
>
> console=ttyS0,38400 console=tty0
>
> This assumes COM1, change ttyS0 to ttyS1 for COM2
>
> Obviously change 38400 baud to whatever works best for you and set the
> other end for the same baud, no parity, 8 bits, XON/XOFF flow control
> only, disable DTR and all other hardware line controls.
>
> Set up minicom or equiv for logging to a file, or have a decent scroll
> buffer set in the terminal and you're good to go. Linux will generally
> try to get messages out over serial even if everything else is dead.

That sounds really proffesional ... I will try to set this thing up as it
may help. Do I need to have an special kernel image or recompile kernel?

BTW is it possible to do it with the network interface? I remember that
kgdb can run on ethX.

Besides that, the problem is that most of the laptops today doesn't have
serial port and you have to use usb bundles or something like this. Then,
what is the possible solution?

Thanks very much.

>
> HTH
>
> Tim

--
Saludos a todos
#
# Contra el terrorismo
#

A. Melinte · External Since: Aug 10, 2005 Posts: 13

> >
> > Obviously change 38400 baud to whatever works best for you and set the
> > other end for the same baud, no parity, 8 bits, XON/XOFF flow control
> > only, disable DTR and all other hardware line controls.

Most lileky you will have to use 9600, I did lost characters at higher
speeds. See also
http://tldp.org/HOWTO/Remote-Serial-Console-HOWTO/configure-kernel-gru...tml#CON

>
> BTW is it possible to do it with the network interface? I remember that
> kgdb can run on ethX.

Besides the serial console, you can set the kernel to dump crash information
to a specially formatted partition or to a remote machine unless you thinker
within the network stuff itself. See
http://www.redhat.com/support/wpapers/redhat/netdump/

The simplest is the serial console. And if you do not get oops information,
this means it is a hardware issue (e.g. power or heat), not a kernel one.

Regards
a.

Tim Southerwood · External Since: Apr 23, 2007 Posts: 113

George wrote:

> Hi,
>
> Tim Southerwood <ts DeleteThis @dionic.net> wrote in
> news:4678f1af$0$645$5a6aecb4@news.aaisp.net.uk:
>
>> George wrote:
>>
>>>
>>> This is what I'm doing now. The problem is that the Oops is printed
>>> and halt the computer. Any tip on how to store this opps?
>>>
>>
>> The easiest way is to wire a spare serial port to another PC via
>> aNull Modem cable, run minicom or equivalent on the other PC and boot
>> your troubled machine with an extra pair of boot parameters:
>>
>> console=ttyS0,38400 console=tty0
>>
>> This assumes COM1, change ttyS0 to ttyS1 for COM2
>>
>> Obviously change 38400 baud to whatever works best for you and set the
>> other end for the same baud, no parity, 8 bits, XON/XOFF flow control
>> only, disable DTR and all other hardware line controls.
>>
>> Set up minicom or equiv for logging to a file, or have a decent scroll
>> buffer set in the terminal and you're good to go. Linux will generally
>> try to get messages out over serial even if everything else is dead.
>
> That sounds really proffesional ...

Heh, not really - it is a standard technique.

> I will try to set this thing up as it
> may help. Do I need to have an special kernel image or recompile kernel?

No, unless you've managed to acquire a kernel that is missing console over
serial and/or the serial drivers.

> BTW is it possible to do it with the network interface? I remember that
> kgdb can run on ethX.

Well, I've never tried this, but there is "netconsole", a kernel driver that
can log printk() over UDP.

Here's an excerpt from netconsole.txt in the kernel source (2.6.15) for your
convenience:

******************************************************
started by Ingo Molnar <mingo DeleteThis @redhat.com>, 2001.09.17
2.6 port and netpoll api by Matt Mackall <mpm DeleteThis @selenic.com>, Sep 9 2003

Please send bug reports to Matt Mackall <mpm DeleteThis @selenic.com>

This module logs kernel printk messages over UDP allowing debugging of
problem where disk logging fails and serial consoles are impractical.

It can be used either built-in or as a module. As a built-in,
netconsole initializes immediately after NIC cards and will bring up
the specified interface as soon as possible. While this doesn't allow
capture of early kernel panics, it does capture most of the boot
process.

It takes a string configuration parameter "netconsole" in the
following format:

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]

where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

Examples:

linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc

or

insmod netconsole netconsole=@/,@10.0.0.2/

Built-in netconsole starts immediately after the TCP stack is
initialized and attempts to bring up the supplied dev at the supplied
address.

The remote host can run either 'netcat -u -l -p <port>' or syslogd.
....
******************************************************

The other way is to use syslog on the problem host, but that method is prone
to miss the very printk()'s that you want as by the time the kernel is that
mashed, the last thing it's going to bother with is scheduling a task to
run.

netconsole might be more reliable, as long as your panics don't originate in
or affect the network layer. That's why serial is usually preferred - it's
a simple robust driver that is fairly unlikely to get buggered up, even in
the worst panics.

> Besides that, the problem is that most of the laptops today doesn't have
> serial port and you have to use usb bundles or something like this. Then,
> what is the possible solution?

USB-serial dongle - they're cheap and there are lots that work with linux.

> Thanks very much.
>

You're welcome Smile

Tim

George · External Since: Jul 16, 2004 Posts: 28

Tim Southerwood <ts.DeleteThis@dionic.net> wrote in
news:46790e3f$0$646$5a6aecb4@news.aaisp.net.uk:

> George wrote:
>
>> Hi,
>>
>> Tim Southerwood <ts.DeleteThis@dionic.net> wrote in
>> news:4678f1af$0$645$5a6aecb4@news.aaisp.net.uk:
>>
>>> George wrote:
>>>
>>>>
>>>> This is what I'm doing now. The problem is that the Oops is printed
>>>> and halt the computer. Any tip on how to store this opps?
>>>>
>>>
>>> The easiest way is to wire a spare serial port to another PC via
>>> aNull Modem cable, run minicom or equivalent on the other PC and
>>> boot your troubled machine with an extra pair of boot parameters:
>>>
>>> console=ttyS0,38400 console=tty0
>>>
>>> This assumes COM1, change ttyS0 to ttyS1 for COM2
>>>
>>> Obviously change 38400 baud to whatever works best for you and set
>>> the other end for the same baud, no parity, 8 bits, XON/XOFF flow
>>> control only, disable DTR and all other hardware line controls.
>>>
>>> Set up minicom or equiv for logging to a file, or have a decent
>>> scroll buffer set in the terminal and you're good to go. Linux will
>>> generally try to get messages out over serial even if everything
>>> else is dead.
>>
>> That sounds really proffesional ...
>
> Heh, not really - it is a standard technique.
>
>> I will try to set this thing up as it
>> may help. Do I need to have an special kernel image or recompile
>> kernel?
>
> No, unless you've managed to acquire a kernel that is missing console
> over serial and/or the serial drivers.
>
>> BTW is it possible to do it with the network interface? I remember
>> that kgdb can run on ethX.
>
> Well, I've never tried this, but there is "netconsole", a kernel
> driver that can log printk() over UDP.
>
> Here's an excerpt from netconsole.txt in the kernel source (2.6.15)
> for your convenience:
>
> ******************************************************
> started by Ingo Molnar <mingo.DeleteThis@redhat.com>, 2001.09.17
> 2.6 port and netpoll api by Matt Mackall <mpm.DeleteThis@selenic.com>, Sep 9 2003
>
> Please send bug reports to Matt Mackall <mpm.DeleteThis@selenic.com>
>
> This module logs kernel printk messages over UDP allowing debugging of
> problem where disk logging fails and serial consoles are impractical.
>
> It can be used either built-in or as a module. As a built-in,
> netconsole initializes immediately after NIC cards and will bring up
> the specified interface as soon as possible. While this doesn't allow
> capture of early kernel panics, it does capture most of the boot
> process.
>
> It takes a string configuration parameter "netconsole" in the
> following format:
>
> netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macadd
> r]
>
> where
> src-port source for UDP packets (defaults to 6665)
> src-ip source IP to use (interface address)
> dev network interface (eth0)
> tgt-port port for logging agent (6666)
> tgt-ip IP address for logging agent
> tgt-macaddr ethernet MAC address for logging agent
> (broadcast)
>
> Examples:
>
> linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
>
> or
>
> insmod netconsole netconsole=@/,@10.0.0.2/
>
> Built-in netconsole starts immediately after the TCP stack is
> initialized and attempts to bring up the supplied dev at the supplied
> address.
>
> The remote host can run either 'netcat -u -l -p <port>' or syslogd.
> ...
> ******************************************************
>
> The other way is to use syslog on the problem host, but that method is
> prone to miss the very printk()'s that you want as by the time the
> kernel is that mashed, the last thing it's going to bother with is
> scheduling a task to run.
>
> netconsole might be more reliable, as long as your panics don't
> originate in or affect the network layer. That's why serial is usually
> preferred - it's a simple robust driver that is fairly unlikely to get
> buggered up, even in the worst panics.
>

Great explanation... I will have a look at it as I haven't got neither
usb-dongle right now nor serial Wink

>> Besides that, the problem is that most of the laptops today doesn't
>> have serial port and you have to use usb bundles or something like
>> this. Then, what is the possible solution?
>
> USB-serial dongle - they're cheap and there are lots that work with
> linux.

After sending the post I start looking and found some Wink

>> Thanks very much.
>>
>
> You're welcome Smile

>
> Tim
>

Thanks again. Let's see if with all that I can find out where the problem
is.

--
Saludos a todos
#
# Contra el terrorismo
#

George · External Since: Jul 16, 2004 Posts: 28

Tim Southerwood <ts.DeleteThis@dionic.net> wrote in
news:46790e3f$0$646$5a6aecb4@news.aaisp.net.uk:

> George wrote:
>
>> Hi,
>>
>> Tim Southerwood <ts.DeleteThis@dionic.net> wrote in
>> news:4678f1af$0$645$5a6aecb4@news.aaisp.net.uk:
>>
>>> George wrote:
>>>
>>>>
>>>> This is what I'm doing now. The problem is that the Oops is printed
>>>> and halt the computer. Any tip on how to store this opps?
>>>>
>>>
>>> The easiest way is to wire a spare serial port to another PC via
>>> aNull Modem cable, run minicom or equivalent on the other PC and
>>> boot your troubled machine with an extra pair of boot parameters:
>>>
>>> console=ttyS0,38400 console=tty0
>>>
>>> This assumes COM1, change ttyS0 to ttyS1 for COM2
>>>
>>> Obviously change 38400 baud to whatever works best for you and set
>>> the other end for the same baud, no parity, 8 bits, XON/XOFF flow
>>> control only, disable DTR and all other hardware line controls.
>>>
>>> Set up minicom or equiv for logging to a file, or have a decent
>>> scroll buffer set in the terminal and you're good to go. Linux will
>>> generally try to get messages out over serial even if everything
>>> else is dead.
>>
>> That sounds really proffesional ...
>
> Heh, not really - it is a standard technique.
>
>> I will try to set this thing up as it
>> may help. Do I need to have an special kernel image or recompile
>> kernel?
>
> No, unless you've managed to acquire a kernel that is missing console
> over serial and/or the serial drivers.
>
>> BTW is it possible to do it with the network interface? I remember
>> that kgdb can run on ethX.
>
> Well, I've never tried this, but there is "netconsole", a kernel
> driver that can log printk() over UDP.
>
> Here's an excerpt from netconsole.txt in the kernel source (2.6.15)
> for your convenience:
>
> ******************************************************
> started by Ingo Molnar <mingo.DeleteThis@redhat.com>, 2001.09.17
> 2.6 port and netpoll api by Matt Mackall <mpm.DeleteThis@selenic.com>, Sep 9 2003
>
> Please send bug reports to Matt Mackall <mpm.DeleteThis@selenic.com>
>
> This module logs kernel printk messages over UDP allowing debugging of
> problem where disk logging fails and serial consoles are impractical.
>
> It can be used either built-in or as a module. As a built-in,
> netconsole initializes immediately after NIC cards and will bring up
> the specified interface as soon as possible. While this doesn't allow
> capture of early kernel panics, it does capture most of the boot
> process.
>
> It takes a string configuration parameter "netconsole" in the
> following format:
>
> netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macadd
> r]
>
> where
> src-port source for UDP packets (defaults to 6665)
> src-ip source IP to use (interface address)
> dev network interface (eth0)
> tgt-port port for logging agent (6666)
> tgt-ip IP address for logging agent
> tgt-macaddr ethernet MAC address for logging agent
> (broadcast)
>
> Examples:
>
> linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
>
> or
>
> insmod netconsole netconsole=@/,@10.0.0.2/
>
> Built-in netconsole starts immediately after the TCP stack is
> initialized and attempts to bring up the supplied dev at the supplied
> address.
>
> The remote host can run either 'netcat -u -l -p <port>' or syslogd.
> ...
> ******************************************************
>
> The other way is to use syslog on the problem host, but that method is
> prone to miss the very printk()'s that you want as by the time the
> kernel is that mashed, the last thing it's going to bother with is
> scheduling a task to run.
>
> netconsole might be more reliable, as long as your panics don't
> originate in or affect the network layer. That's why serial is usually
> preferred - it's a simple robust driver that is fairly unlikely to get
> buggered up, even in the worst panics.
>
>> Besides that, the problem is that most of the laptops today doesn't
>> have serial port and you have to use usb bundles or something like
>> this. Then, what is the possible solution?
>
> USB-serial dongle - they're cheap and there are lots that work with
> linux.
>
>> Thanks very much.
>>
>
> You're welcome Smile

>
> Tim

Thanks very much... The netconsole works perfectly... At least now I can
see the whole kernel panic, although still have to familiarize with the
ksymoops and its output.

See you

Jean-François · External Since: May 11, 2004 Posts: 10

George <george.news DeleteThis @NOSPAMgmx.net> wrote:
> Hi all,

Hi George,

[snip]
> I've been reading for debugging tools and some time ago I used kgdb, but it
> is not available for kernel 2.6.20.
[snip]

I made a new version of the simple 'icebox' kernel debugger created by
David Moore that works with the kernel 2.6.21 (see my page below).
It is basic, but very usable. Its main advantage is that you don't
need any kernel patch: it is build as any foreign lambda module.

Cheers.

--
Ken ar c'hentañ | ** Breizh ha Linux atav! **
Jef | http://moinejf.free.fr/