cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom..

Justin Mattock · External Since: Apr 14, 2009 Posts: 12

Hello,
I'm not sure how to handle this,
while compiling firefox-3.6b1.source
I get this with the default compiling options,
as well as custom:

[ 532.942324] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
[ 532.942330] Pid: 16002, comm: cc1plus Tainted: P
2.6.32-rc5-00083-g04ea458 #2
[ 532.942333] Call Trace:
[ 532.942342] [<ffffffff810bce11>] T.417+0x7c/0x245
[ 532.942347] [<ffffffff810bd11c>] __out_of_memory+0x142/0x159
[ 532.942352] [<ffffffff810bd1a1>] out_of_memory+0x6e/0x9d
[ 532.942357] [<ffffffff810c0086>] __alloc_pages_nodemask+0x47e/0x5cc
[ 532.942363] [<ffffffff810d2115>] handle_mm_fault+0x25d/0x68e
[ 532.942369] [<ffffffff813dfc1d>] do_page_fault+0x2bb/0x2d3
[ 532.942373] [<ffffffff813ddb25>] page_fault+0x25/0x30
[ 532.942376] Mem-Info:
[ 532.942378] DMA per-cpu:
[ 532.942380] CPU 0: hi: 0, btch: 1 usd: 0
[ 532.942383] CPU 1: hi: 0, btch: 1 usd: 0
[ 532.942385] DMA32 per-cpu:
[ 532.942388] CPU 0: hi: 186, btch: 31 usd: 94
[ 532.942391] CPU 1: hi: 186, btch: 31 usd: 23
[ 532.942393] Normal per-cpu:
[ 532.942395] CPU 0: hi: 186, btch: 31 usd: 150
[ 532.942398] CPU 1: hi: 186, btch: 31 usd: 155
[ 532.942404] active_anon:707575 inactive_anon:264673 isolated_anon:0
[ 532.942406] active_file:58 inactive_file:33 isolated_file:0
[ 532.942407] unevictable:0 dirty:0 writeback:0 unstable:0 buffer:71
[ 532.942408] free:6998 slab_reclaimable:2697 slab_unreclaimable:16267
[ 532.942409] mapped:136 shmem:64 pagetables:2761 bounce:0
[ 532.942418] DMA free:15944kB min:28kB low:32kB high:40kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15360kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[ 532.942425] lowmem_reserve[]: 0 2976 3986 3986
[ 532.942436] DMA32 free:10020kB min:6020kB low:7524kB high:9028kB
active_anon:2360492kB inactive_anon:590196kB active_file:84kB
inactive_file:68kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:3047792kB mlocked:0kB dirty:0kB
writeback:0kB mapped:136kB shmem:80kB slab_reclaimable:0kB
slab_unreclaimable:88kB kernel_stack:0kB pagetables:6100kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448
all_unreclaimable? yes
[ 532.942443] lowmem_reserve[]: 0 0 1010 1010
[ 532.942454] Normal free:2028kB min:2040kB low:2548kB high:3060kB
active_anon:469808kB inactive_anon:468496kB active_file:148kB
inactive_file:64kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:1034240kB mlocked:0kB dirty:0kB
writeback:0kB mapped:408kB shmem:176kB slab_reclaimable:10788kB
slab_unreclaimable:64972kB kernel_stack:800kB pagetables:4944kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:413
all_unreclaimable? yes
[ 532.942461] lowmem_reserve[]: 0 0 0 0
[ 532.942465] DMA: 2*4kB 2*8kB 3*16kB 2*32kB 3*64kB 2*128kB 2*256kB
1*512kB 2*1024kB 2*2048kB 2*4096kB = 15944kB
[ 532.942478] DMA32: 7*4kB 7*8kB 5*16kB 4*32kB 2*64kB 1*128kB 1*256kB
4*512kB 3*1024kB 0*2048kB 1*4096kB = 10020kB
[ 532.942490] Normal: 507*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2028kB
[ 532.942501] 7974 total pagecache pages
[ 532.942503] 7778 pages in swap cache
[ 532.942506] Swap cache stats: add 112290, delete 104512, find 3595/4051
[ 532.942508] Free swap = 0kB
[ 532.942510] Total swap = 431632kB
[ 532.957941] 1048576 pages RAM
[ 532.957943] 40518 pages reserved
[ 532.957945] 312 pages shared
[ 532.957947] 1000176 pages non-shared
[ 532.957951] Out of memory: kill process 16001 (c++) score 543727 or a child
[ 532.957955] Killed process 16002 (cc1plus)

I just compiled the latest gcc snapshot a few days
ago.

--
Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Justin P. Mattock · External Since: Jan 23, 2009 Posts: 29

Jiri Slaby wrote:
> On 11/02/2009 11:02 PM, Justin Mattock wrote:
>
>> Now with this oom-killer I'm
>> hitting this on an imac9,1 with
>> gcc (GCC) 4.5.0 20091029 (experimental)
>>
>
> So there is probably a leak in the gcc chain. Does this happen with a
> stable gcc version?
>
>
well right now I'(on the macbook), seems to be going for a good ten
minutes before
I hit an error with a header file(nss)

I'll see if I can get this thing to compile all the way through.
(knock on wood) then go from there.

Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

KOSAKI Motohiro · External Since: Jan 15, 2009 Posts: 112

> Hello,
> I'm not sure how to handle this,
> while compiling firefox-3.6b1.source
> I get this with the default compiling options,
> as well as custom:
>
>
> [ 532.942324] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
> [ 532.942330] Pid: 16002, comm: cc1plus Tainted: P
> 2.6.32-rc5-00083-g04ea458 #2
> [ 532.942333] Call Trace:
> [ 532.942342] [<ffffffff810bce11>] T.417+0x7c/0x245
> [ 532.942347] [<ffffffff810bd11c>] __out_of_memory+0x142/0x159
> [ 532.942352] [<ffffffff810bd1a1>] out_of_memory+0x6e/0x9d
> [ 532.942357] [<ffffffff810c0086>] __alloc_pages_nodemask+0x47e/0x5cc
> [ 532.942363] [<ffffffff810d2115>] handle_mm_fault+0x25d/0x68e
> [ 532.942369] [<ffffffff813dfc1d>] do_page_fault+0x2bb/0x2d3
> [ 532.942373] [<ffffffff813ddb25>] page_fault+0x25/0x30
> [ 532.942376] Mem-Info:
> [ 532.942378] DMA per-cpu:
> [ 532.942380] CPU 0: hi: 0, btch: 1 usd: 0
> [ 532.942383] CPU 1: hi: 0, btch: 1 usd: 0
> [ 532.942385] DMA32 per-cpu:
> [ 532.942388] CPU 0: hi: 186, btch: 31 usd: 94
> [ 532.942391] CPU 1: hi: 186, btch: 31 usd: 23
> [ 532.942393] Normal per-cpu:
> [ 532.942395] CPU 0: hi: 186, btch: 31 usd: 150
> [ 532.942398] CPU 1: hi: 186, btch: 31 usd: 155
> [ 532.942404] active_anon:707575 inactive_anon:264673 isolated_anon:0
> [ 532.942406] active_file:58 inactive_file:33 isolated_file:0

file cache (active_file + inactive_file) was very little. It indicate anyone waste too much memory.
I doubt you use buggy compiler.

> [ 532.942407] unevictable:0 dirty:0 writeback:0 unstable:0 buffer:71
> [ 532.942408] free:6998 slab_reclaimable:2697 slab_unreclaimable:16267
> [ 532.942409] mapped:136 shmem:64 pagetables:2761 bounce:0
> [ 532.942418] DMA free:15944kB min:28kB low:32kB high:40kB
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15360kB
> mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB
> pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? yes
> [ 532.942425] lowmem_reserve[]: 0 2976 3986 3986
> [ 532.942436] DMA32 free:10020kB min:6020kB low:7524kB high:9028kB
> active_anon:2360492kB inactive_anon:590196kB active_file:84kB
> inactive_file:68kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:3047792kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:136kB shmem:80kB slab_reclaimable:0kB
> slab_unreclaimable:88kB kernel_stack:0kB pagetables:6100kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448
> all_unreclaimable? yes
> [ 532.942443] lowmem_reserve[]: 0 0 1010 1010
> [ 532.942454] Normal free:2028kB min:2040kB low:2548kB high:3060kB
> active_anon:469808kB inactive_anon:468496kB active_file:148kB
> inactive_file:64kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:1034240kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:408kB shmem:176kB slab_reclaimable:10788kB
> slab_unreclaimable:64972kB kernel_stack:800kB pagetables:4944kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:413
> all_unreclaimable? yes
> [ 532.942461] lowmem_reserve[]: 0 0 0 0
> [ 532.942465] DMA: 2*4kB 2*8kB 3*16kB 2*32kB 3*64kB 2*128kB 2*256kB
> 1*512kB 2*1024kB 2*2048kB 2*4096kB = 15944kB
> [ 532.942478] DMA32: 7*4kB 7*8kB 5*16kB 4*32kB 2*64kB 1*128kB 1*256kB
> 4*512kB 3*1024kB 0*2048kB 1*4096kB = 10020kB
> [ 532.942490] Normal: 507*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
> 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2028kB
> [ 532.942501] 7974 total pagecache pages
> [ 532.942503] 7778 pages in swap cache
> [ 532.942506] Swap cache stats: add 112290, delete 104512, find 3595/4051
> [ 532.942508] Free swap = 0kB
> [ 532.942510] Total swap = 431632kB
> [ 532.957941] 1048576 pages RAM
> [ 532.957943] 40518 pages reserved
> [ 532.957945] 312 pages shared
> [ 532.957947] 1000176 pages non-shared
> [ 532.957951] Out of memory: kill process 16001 (c++) score 543727 or a child
> [ 532.957955] Killed process 16002 (cc1plus)
>
> I just compiled the latest gcc snapshot a few days
> ago.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Justin P. Mattock · External Since: Jan 23, 2009 Posts: 29

KOSAKI Motohiro wrote:
>> Hello,
>> I'm not sure how to handle this,
>> while compiling firefox-3.6b1.source
>> I get this with the default compiling options,
>> as well as custom:
>>
>>
>> [ 532.942324] cc1plus invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
>> [ 532.942330] Pid: 16002, comm: cc1plus Tainted: P
>> 2.6.32-rc5-00083-g04ea458 #2
>> [ 532.942333] Call Trace:
>> [ 532.942342] [<ffffffff810bce11>] T.417+0x7c/0x245
>> [ 532.942347] [<ffffffff810bd11c>] __out_of_memory+0x142/0x159
>> [ 532.942352] [<ffffffff810bd1a1>] out_of_memory+0x6e/0x9d
>> [ 532.942357] [<ffffffff810c0086>] __alloc_pages_nodemask+0x47e/0x5cc
>> [ 532.942363] [<ffffffff810d2115>] handle_mm_fault+0x25d/0x68e
>> [ 532.942369] [<ffffffff813dfc1d>] do_page_fault+0x2bb/0x2d3
>> [ 532.942373] [<ffffffff813ddb25>] page_fault+0x25/0x30
>> [ 532.942376] Mem-Info:
>> [ 532.942378] DMA per-cpu:
>> [ 532.942380] CPU 0: hi: 0, btch: 1 usd: 0
>> [ 532.942383] CPU 1: hi: 0, btch: 1 usd: 0
>> [ 532.942385] DMA32 per-cpu:
>> [ 532.942388] CPU 0: hi: 186, btch: 31 usd: 94
>> [ 532.942391] CPU 1: hi: 186, btch: 31 usd: 23
>> [ 532.942393] Normal per-cpu:
>> [ 532.942395] CPU 0: hi: 186, btch: 31 usd: 150
>> [ 532.942398] CPU 1: hi: 186, btch: 31 usd: 155
>> [ 532.942404] active_anon:707575 inactive_anon:264673 isolated_anon:0
>> [ 532.942406] active_file:58 inactive_file:33 isolated_file:0
>>
>
> file cache (active_file + inactive_file) was very little. It indicate anyone waste too much memory.
> I doubt you use buggy compiler.
>
>
>
>
hmm... then this is something with firefox then..
In that case I'll continue to build my system with
this compiler.
Although a bit concerned building everything
for the system with a compiler that shows some
memory issue,but if you say its not the compiler,
then I'll carry on with what I'm doing.
(and use an older compiler for firefox).

Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Andrew Morton · External Since: Feb 02, 2007 Posts: 2303

On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock <justinmattock.RemoveThis@gmail.com> wrote:

> Hello,
> I'm not sure how to handle this,
> while compiling firefox-3.6b1.source
> I get this with the default compiling options,
> as well as custom:
>
> ...
>
> active_anon:2360492kB inactive_anon:590196kB active_file:84kB

2.8GB of anonymous memory

> [ 532.942508] Free swap = 0kB
> [ 532.942510] Total swap = 431632kB

430MB of swap, all used up.

That's a genuine OOM. Something (presumably cc1plus) has consumed
waaaay too much memory, quite possibly leaked it.

It would help if the oom-killer were to print some information about
the oom-killed process's memory footprint.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Justin P. Mattock · External Since: Jan 23, 2009 Posts: 29

Andrew Morton wrote:
> On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock<justinmattock.RemoveThis@gmail.com> wrote:
>
>
>> Hello,
>> I'm not sure how to handle this,
>> while compiling firefox-3.6b1.source
>> I get this with the default compiling options,
>> as well as custom:
>>
>> ...
>>
>> active_anon:2360492kB inactive_anon:590196kB active_file:84kB
>>
>
> 2.8GB of anonymous memory
>
>
figured it would be good enough
(I think I have 4gig's total)
>> [ 532.942508] Free swap = 0kB
>> [ 532.942510] Total swap = 431632kB
>>
>
> 430MB of swap, all used up.
>
>
yep, narrow down to the smallest amount.
> That's a genuine OOM. Something (presumably cc1plus) has consumed
> waaaay too much memory, quite possibly leaked it.
>
> It would help if the oom-killer were to print some information about
> the oom-killed process's memory footprint.
>
>
>
I still have everything setup(if you need me to add a debug patch
let me know)
as for compiling: libc compiled fine, kernel fine,
and every package on the clfs list up to boot up the fresh system.
(was figuring out how to compiling/install firefox before
I threw the old system away).

stable gcc(4.4*) on the macbook(same os/kernel) compiled fine
firefox, xulrunner, and in the process thunderbird...

Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Jiri Slaby · External Since: Nov 04, 2006 Posts: 645

On 11/04/2009 07:44 AM, Justin P. Mattock wrote:
> as for compiling: libc compiled fine, kernel fine,
> and every package on the clfs list up to boot up the fresh system.

It might be pretty c++ only, I think.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

KOSAKI Motohiro · External Since: Jan 15, 2009 Posts: 112

> On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock <justinmattock.RemoveThis@gmail.com> wrote:
>
> > Hello,
> > I'm not sure how to handle this,
> > while compiling firefox-3.6b1.source
> > I get this with the default compiling options,
> > as well as custom:
> >
> > ...
> >
> > active_anon:2360492kB inactive_anon:590196kB active_file:84kB
>
> 2.8GB of anonymous memory
>
> > [ 532.942508] Free swap = 0kB
> > [ 532.942510] Total swap = 431632kB
>
> 430MB of swap, all used up.
>
> That's a genuine OOM. Something (presumably cc1plus) has consumed
> waaaay too much memory, quite possibly leaked it.
>
> It would help if the oom-killer were to print some information about
> the oom-killed process's memory footprint.
>

How about this?

========
Subject: [PATCH] oom: show vsz and rss information of the killed process

In typical oom anylysis scenario, we frequently want to know the killed
process has memory leak or not at first step.
This patch add vsz and rss information to oom log for helping its
analysis. It save much times of debugging guys.

example:
===================================================================
rsyslogd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Pid: 1308, comm: rsyslogd Not tainted 2.6.32-rc6 #24
Call Trace:
[<ffffffff8132e35b>] ?_spin_unlock+0x2b/0x40
[<ffffffff810f186e>] oom_kill_process+0xbe/0x2b0

(snip)

492283 pages non-shared
Out of memory: kill process 2341 (memhog) score 527276 or a child
Killed process 2341 (memhog) vsz:1054552kB, anon-rss:970588kB, file-rss:4kB
===========================================================================
^
|
here

Signed-off-by: KOSAKI Motohiro <kosaki.motohiro.RemoveThis@jp.fujitsu.com>
---
mm/oom_kill.c | 15 ++++++++++++---
1 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index ea2147d..498e6f6 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -337,6 +337,8 @@ static void dump_tasks(const struct mem_cgroup *mem)
} while_each_thread(g, p);
}

+#define K(x) ((x) << (PAGE_SHIFT-10))
+
/*
* Send SIGKILL to the selected process irrespective of CAP_SYS_RAW_IO
* flag though it's unlikely that we select a process with CAP_SYS_RAW_IO
@@ -356,9 +358,16 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
return;
}

- if (verbose)
- printk(KERN_ERR "Killed process %d (%s)\n",
- task_pid_nr(p), p->comm);
+ if (verbose) {
+ task_lock(p);
+ printk(KERN_ERR "Killed process %d (%s) "
+ "vsz:%lukB, anon-rss:%lukB, file-rss:%lukB\n",
+ task_pid_nr(p), p->comm,
+ K(p->mm->total_vm),
+ K(get_mm_counter(p->mm, anon_rss)),
+ K(get_mm_counter(p->mm, file_rss)));
+ task_unlock(p);
+ }

/*
* We give our sacrificial lamb high priority and access to
--
1.6.2.5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Dave Korn · External Since: Nov 04, 2009 Posts: 5

Andrew Morton wrote:
> On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock <justinmattock.RemoveThis@gmail.com> wrote:
>
>> Hello,
>> I'm not sure how to handle this,
>> while compiling firefox-3.6b1.source
>> I get this with the default compiling options,
>> as well as custom:
>>
>> ...
>>
>> active_anon:2360492kB inactive_anon:590196kB active_file:84kB
>
> 2.8GB of anonymous memory
>
>> [ 532.942508] Free swap = 0kB
>> [ 532.942510] Total swap = 431632kB
>
> 430MB of swap, all used up.
>
> That's a genuine OOM. Something (presumably cc1plus) has consumed
> waaaay too much memory, quite possibly leaked it.
>
> It would help if the oom-killer were to print some information about
> the oom-killed process's memory footprint.

I would think that the quickest way to proceed would be to re-run the
failing compile command under gdb at the command-line and see what it's doing
when the oom killer signals it, wouldn't it? Or turn up the swap until it
doesn't get killed and see what info can be gleaned from the cc1(plus?)
-fmem-report output.

cheers,
DaveK

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Justin P. Mattock · External Since: Jan 23, 2009 Posts: 29

Dave Korn wrote:
> Andrew Morton wrote:
>
>> On Mon, 2 Nov 2009 13:29:29 -0800 Justin Mattock<justinmattock.DeleteThis@gmail.com> wrote:
>>
>>
>>> Hello,
>>> I'm not sure how to handle this,
>>> while compiling firefox-3.6b1.source
>>> I get this with the default compiling options,
>>> as well as custom:
>>>
>>> ...
>>>
>>> active_anon:2360492kB inactive_anon:590196kB active_file:84kB
>>>
>> 2.8GB of anonymous memory
>>
>>
>>> [ 532.942508] Free swap = 0kB
>>> [ 532.942510] Total swap = 431632kB
>>>
>> 430MB of swap, all used up.
>>
>> That's a genuine OOM. Something (presumably cc1plus) has consumed
>> waaaay too much memory, quite possibly leaked it.
>>
>> It would help if the oom-killer were to print some information about
>> the oom-killed process's memory footprint.
>>
>
> I would think that the quickest way to proceed would be to re-run the
> failing compile command under gdb at the command-line and see what it's doing
> when the oom killer signals it, wouldn't it? Or turn up the swap until it
> doesn't get killed and see what info can be gleaned from the cc1(plus?)
> -fmem-report output.
>
> cheers,
> DaveK
>
>
>
I can try, only issue I have is I don't
use a distro, so building anything requires me
to hand compile it(hopefully not difficult for gdb).

So give me some time on this and I'll see if I can get this up
and running, and add that patch to kernel then go from there.

Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Andrew Morton · External Since: Feb 02, 2007 Posts: 2303

On Wed, 4 Nov 2009 18:32:16 +0900 (JST) KOSAKI Motohiro <kosaki.motohiro.RemoveThis@jp.fujitsu.com> wrote:

> > It would help if the oom-killer were to print some information about
> > the oom-killed process's memory footprint.
> >
>
>
> How about this?

looks good, thanks.

> ========
> Subject: [PATCH] oom: show vsz and rss information of the killed process
>
> In typical oom anylysis scenario, we frequently want to know the killed
> process has memory leak or not at first step.
> This patch add vsz and rss information to oom log for helping its
> analysis. It save much times of debugging guys.
>
> example:
> ===================================================================
> rsyslogd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
> Pid: 1308, comm: rsyslogd Not tainted 2.6.32-rc6 #24
> Call Trace:
> [<ffffffff8132e35b>] ?_spin_unlock+0x2b/0x40
> [<ffffffff810f186e>] oom_kill_process+0xbe/0x2b0
>
> (snip)
>
> 492283 pages non-shared
> Out of memory: kill process 2341 (memhog) score 527276 or a child
> Killed process 2341 (memhog) vsz:1054552kB, anon-rss:970588kB, file-rss:4kB
> ===========================================================================
> ^
> |
> here
> ...
>
> + if (verbose) {
> + task_lock(p);

We need to be careful with which locks we take on the oom-killer path,
because it can be called by code which already holds locks. But I
expect task_lock() will be OK.

> + printk(KERN_ERR "Killed process %d (%s) "
> + "vsz:%lukB, anon-rss:%lukB, file-rss:%lukB\n",
> + task_pid_nr(p), p->comm,
> + K(p->mm->total_vm),
> + K(get_mm_counter(p->mm, anon_rss)),
> + K(get_mm_counter(p->mm, file_rss)));
> + task_unlock(p);
> + }

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Dave Korn · External Since: Nov 04, 2009 Posts: 5

Justin P. Mattock wrote:

> I can try, only issue I have is I don't
> use a distro, so building anything requires me
> to hand compile it

Oh, ouch!

> (hopefully not difficult for gdb).

Indeed, hopefully not.

> So give me some time on this and I'll see if I can get this up
> and running, and add that patch to kernel then go from there.

The one thing you can still try straight away for minimal effort is the
-fmem-report option, but it's also the least informative...

cheers,
DaveK

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Dave Korn · External Since: Nov 04, 2009 Posts: 5

Justin Mattock wrote:
> here's what I did:
> valgrind --tool=memcheck --leak-check=full -v make -f client.mk build

> ==4072== LEAK SUMMARY:

> I'll try out gdb, and more of valgrind.

Yep, that doesn't tell us a lot in its default modes. I'm not a valgrind
expert but it looks from the docs like you want to try the Massif tool: it
looks really thorough.

http://valgrind.org/docs/manual/ms-manual.html

cheers,
DaveK

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Dave Korn · External Since: Nov 04, 2009 Posts: 5

Justin Mattock wrote:

> O.k. here is the info from dmesg(with the patch added)
> and what -fmem-report:

I don't know how to read the oom dmesg, but as to the -fmem-report:

> Memory still allocated at the end of the compilation process
> Size Allocated Used Overhead
> Total 7200k 5293k 104k

.... what that's telling us is that there isn't a substantial leak in GCC, as
there's only 7 meg left unreclaimed by GC at the end. I think we'll have to
wait and see what the debugger tells us; either GCC really is using that much
memory in processing the file, or there's some kind of system or kernel bug
you're running into that is causing a leak in the VMM rather than the application.

> String pool

> bytes 86k (17592186044415M overhead)

0xFFFFFFFFFFF00000, lol, wut? It's possible that indicates some sort of
memory corruption going on. Maybe valgrind can help, do you have that?

cheers,
DaveK

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

KOSAKI Motohiro · External Since: Jan 15, 2009 Posts: 112

> > + if (verbose) {
> > + task_lock(p);
>
> We need to be careful with which locks we take on the oom-killer path,
> because it can be called by code which already holds locks. But I
> expect task_lock() will be OK.

Sure.
task_lock() is already used various oom path. I think this is ok.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Dave Korn · External Since: Nov 04, 2009 Posts: 5

Justin Mattock wrote:

> ==1830== Command: c++ -o jsxml.o -c -DOSTYPE="Linux2.6" -DOSARCH=Linux

Ah, you're running it on the "c++" utility and it's reporting the stats for
that, but how it works is that "c++" (and "gcc", "g++", et al) is just a
driver, that parses the command line arguments and shells out to the actual
compiler ("cc1plus"), assembler and linker to get them to do all the work.

If you add "-v --save-temps" to the c++ invocation, it'll show you the
separate command lines it executes for the subprograms; the first invocation
will be of cc1plus, using the -E flag to generate the preprocessed source into
a .ii file, it's the second invocation you want, the one which uses the
"-fpreprocessed" flag and names the .ii file as input, which is the one that
actually then compiles the pre-processed source into assembly. For fuller
explanation, see the GCC wiki:

http://gcc.gnu.org/wiki/DebuggingGCC

cheers,
DaveK

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Justin Mattock · External Since: Apr 14, 2009 Posts: 12

On Wed, Nov 4, 2009 at 4:36 PM, Dave Korn
<dave.korn.cygwin.TakeThisOut@googlemail.com> wrote:
> Justin Mattock wrote:
>> here's what I did:
>> valgrind --tool=memcheck --leak-check=full -v make -f client.mk build
>
>> ==4072== LEAK SUMMARY:
>
>> I'll try out gdb, and more of valgrind.
>
> Yep, that doesn't tell us a lot in its default modes. I'm not a valgrind
> expert but it looks from the docs like you want to try the Massif tool: it
> looks really thorough.
>
> http://valgrind.org/docs/manual/ms-manual.html
>
> cheers,
> DaveK
>
>
>

o.k. round2 I can try gdb(now that I have somewhat of an idea of
how to use one of these tools).

in the meanwhile hopefully this is useful(if not just clip, and I'll try
to gather something useful)

using this for valgrind and the command(below in log)
valgrind -v --tool=memcheck --leak-check=yes --num-callers=10
--leak-check=full --show-reachable=yes --gen-suppressions=yes
(log of valgrind is a bit long, so here's just a few of what
suppresions printed out:)

==1830== Memcheck, a memory error detector
==1830== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==1830== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==1830== Command: c++ -o jsxml.o -c -DOSTYPE="Linux2.6" -DOSARCH=Linux
-DEXPORT_JS_API -DJS_USE_SAFE_ARENA
-I/home/justin/LFS/firefox/mozilla-1.9.2/js/src -I.
-I./../../dist/include -I./../../dist/include/nsprpub
-I/usr/include/nspr -I/home/justin/LFS/firefox/mozilla-1.9.2/js/src
-fPIC -fno-rtti -fno-exceptions -Wall -Wpointer-arith
-Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy
-Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof
-Wno-variadic-macros -Wno-long-long -pedantic -fno-strict-aliasing
-pthread -pipe -DNDEBUG -DTRIMMED -m64 -mtune=core2 -march=core2 -O2
-pipe -fomit-frame-pointer -DMOZILLA_CLIENT -include ./js-confdefs.h
-Wp,-MD,.deps/jsxml.pp
/home/justin/LFS/firefox/mozilla-1.9.2/js/src/jsxml.cpp
==1830==
--1830-- Valgrind options:
--1830-- -v
--1830-- --tool=memcheck
--1830-- --leak-check=yes
--1830-- --num-callers=10
--1830-- --leak-check=full
--1830-- --show-reachable=yes
--1830-- --gen-suppressions=yes
--1830-- Contents of /proc/version:
--1830-- Linux version 2.6.32-rc5-00083-g04ea458 (justin@Linux-0)
(gcc version 4.4.1 (GCC for Cross-LFS 4.4.1.20090722) ) #2 SMP Sat Oct
24 21:38:54 PDT 2009
--1830-- Arch and hwcaps: AMD64, amd64-sse3-cx16
--1830-- Page sizes: currently 4096, max supported 4096
--1830-- Valgrind library directory: /usr/lib/valgrind
--1830-- Reading syms from /usr/bin/c++ (0x400000)
--1830-- Reading syms from /lib/ld-2.10.90.so (0x4000000)
--1830-- Reading syms from /usr/lib/valgrind/memcheck-amd64-linux (0x38000000)
--1830-- object doesn't have a dynamic symbol table
--1830-- Reading suppressions file: /usr/lib/valgrind/default.supp
--1830-- REDIR: 0x4016110 (strlen) redirected to 0x38040607
(vgPlain_amd64_linux_REDIR_FOR_strlen)
--1830-- Reading syms from
/usr/lib/valgrind/vgpreload_core-amd64-linux.so (0x4a20000)
--1830-- Reading syms from
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so (0x4c21000)
==1830== WARNING: new redirection conflicts with existing -- ignoring it
--1830-- new: 0x04016110 (strlen ) R-> 0x04c24fd0 strlen
--1830-- REDIR: 0x4015f30 (index) redirected to 0x4c24d50 (index)
--1830-- REDIR: 0x40160e0 (strcmp) redirected to 0x4c252e0 (strcmp)
--1830-- Reading syms from /lib/libc-2.10.90.so (0x4e28000)
--1830-- REDIR: 0x4ea4c40 (rindex) redirected to 0x4c24bb0 (rindex)
--1830-- REDIR: 0x4e9e010 (malloc) redirected to 0x4c24808 (malloc)
--1830-- REDIR: 0x4ea3330 (strncmp) redirected to 0x4c25210 (strncmp)
--1830-- REDIR: 0x4ea7670 (memcpy) redirected to 0x4c253e0 (memcpy)
--1830-- REDIR: 0x4e9df30 (free) redirected to 0x4c23b44 (free)
--1830-- REDIR: 0x4ea17d0 (strcmp) redirected to 0x4c25280 (strcmp)
--1830-- REDIR: 0x4ea3170 (strlen) redirected to 0x4c24f90 (strlen)
--1830-- REDIR: 0x4ea1750 (index) redirected to 0x4c24c50 (index)
--1830-- REDIR: 0x4ea9da0 (strchrnul) redirected to 0x4c25ff0 (strchrnul)
--1830-- REDIR: 0x4ea6d30 (mempcpy) redirected to 0x4c260e0 (mempcpy)
--1830-- REDIR: 0x4ea5bc0 (memchr) redirected to 0x4c253a0 (memchr)
--1830-- REDIR: 0x4e9f060 (realloc) redirected to 0x4c248b9 (realloc)
--1830-- REDIR: 0x4ea31c0 (strnlen) redirected to 0x4c24f60 (strnlen)
--1830-- REDIR: 0x4ea7360 (stpcpy) redirected to 0x4c25c10 (stpcpy)
--1830-- REDIR: 0x4ea2c20 (strcpy) redirected to 0x4c24ff0 (strcpy)
--1830-- REDIR: 0x4ea9d50 (rawmemchr) redirected to 0x4c26020 (rawmemchr)
--1830-- REDIR: 0x4e9d5d0 (calloc) redirected to 0x4c2322c (calloc)
--1830-- REDIR: 0x4ea4b80 (strncpy) redirected to 0x4c250c0 (strncpy)
--1830-- REDIR: 0x4ea1590 (strcat) redirected to 0x4c24d90 (strcat)
--1830-- REDIR: 0x4e5bf40 (putenv) redirected to 0x4c263d0 (putenv)
{standard input}: Assembler messages:
{standard input}:271839: Warning: end of file not at end of a line;
newline inserted
{standard input}:271896: Error: suffix or operands invalid for `movq'
{standard input}:271896: Error: open CFI at the end of file; missing
..cfi_endproc directive
c++: Internal error: Killed (program cc1plus)
Please submit a full bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
==1830==
==1830== HEAP SUMMARY:
==1830== in use at exit: 25,444 bytes in 75 blocks
==1830== total heap usage: 212 allocs, 137 frees, 36,791 bytes allocated
==1830==
==1830== Searching for pointers to 75 not-freed blocks
==1830== Checked 74,760 bytes
==1830==
==1830== 1 bytes in 1 blocks are still reachable in loss record 1 of 59
==1830== at 0x4C232DA: calloc (vg_replace_malloc.c:418)
==1830== by 0x41BE49: xcalloc (xmalloc.c:162)
==1830== by 0x40F5D6: main (gcc.c:7270)
==1830==
==1830==
==1830== ---- Print suppression ? --- [Return/N/n/Y/y/C/c] ---- y
{
<insert_a_suppression_name_here>
Memcheck:Leak
fun:calloc
fun:xcalloc
fun:main
}
==1830== 4 bytes in 1 blocks are still reachable in loss record 2 of 59
==1830== at 0x4C2488A: malloc (vg_replace_malloc.c:195)
==1830== by 0x41BDF7: xmalloc (xmalloc.c:147)
==1830== by 0x402255: save_string (gcc.c:7660)
==1830== by 0x4034ED: add_preprocessor_option (gcc.c:3447)
==1830== by 0x408A06: process_command (gcc.c:3943)
==1830== by 0x40DE6F: main (gcc.c:6871)
==1830==
==1830==
==1830== ---- Print suppression ? --- [Return/N/n/Y/y/C/c] ---- y
{
<insert_a_suppression_name_here>
Memcheck:Leak
fun:malloc
fun:xmalloc
fun:save_string
fun:add_preprocessor_option
fun:process_command
fun:main
}
==1830== 6 bytes in 1 blocks are still reachable in loss record 3 of 59
==1830== at 0x4C2488A: malloc (vg_replace_malloc.c:195)
==1830== by 0x41BDF7: xmalloc (xmalloc.c:147)
==1830== by 0x41BF11: xstrdup (xstrdup.c:34)
==1830== by 0x411F75: update_path (prefix.c:273)
==1830== by 0x403631: add_prefix (gcc.c:2871)
==1830== by 0x40E1CF: main (gcc.c:7043)
==1830==
==1830==
==1830== ---- Print suppression ? --- [Return/N/n/Y/y/C/c] ---- y
{
<insert_a_suppression_name_here>
Memcheck:Leak
fun:malloc
fun:xmalloc
fun:xstrdup
fun:update_path
fun:add_prefix
fun:main
}
==1830== 7 bytes in 1 blocks are still reachable in loss record 4 of 59
==1830== at 0x4C2488A: malloc (vg_replace_malloc.c:195)
==1830== by 0x41BDF7: xmalloc (xmalloc.c:147)
==1830== by 0x40E64E: main (gcc.c:8236)
==1830==
==1830==
==1830== ---- Print suppression ? --- [Return/N/n/Y/y/C/c] ---- y
{
<insert_a_suppression_name_here>
Memcheck:Leak
fun:malloc
fun:xmalloc
fun:main
}

all I have to do is go into the directory
obj-x86_64-unknown-linux-gnu/js/src/
of mozillas source tree and use valgrind c++ ***
(I'll see what gdb does)

--
Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Justin P. Mattock · External Since: Jan 23, 2009 Posts: 29

Dave Korn wrote:
> Justin Mattock wrote:
>
>
>> O.k. here is the info from dmesg(with the patch added)
>> and what -fmem-report:
>>
>
> I don't know how to read the oom dmesg, but as to the -fmem-report:
>
>
>> Memory still allocated at the end of the compilation process
>> Size Allocated Used Overhead
>> Total 7200k 5293k 104k
>>
>
> ... what that's telling us is that there isn't a substantial leak in GCC, as
> there's only 7 meg left unreclaimed by GC at the end. I think we'll have to
> wait and see what the debugger tells us; either GCC really is using that much
> memory in processing the file, or there's some kind of system or kernel bug
> you're running into that is causing a leak in the VMM rather than the application.
>
>
just finished compiling and installing gdb/valgrind
>> String pool
>>
>
>
>> bytes 86k (17592186044415M overhead)
>>
>
> 0xFFFFFFFFFFF00000, lol, wut? It's possible that indicates some sort of
> memory corruption going on. Maybe valgrind can help, do you have that?
>
> cheers,
> DaveK
>
>
>
Not sure how to use these.(need to read)
Any quick commands I can do to get the info
to you?

Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Justin P. Mattock · External Since: Jan 23, 2009 Posts: 29

Dave Korn wrote:
> Justin Mattock wrote:
>
>
>
>> ==1830== Command: c++ -o jsxml.o -c -DOSTYPE="Linux2.6" -DOSARCH=Linux
>>
>
> Ah, you're running it on the "c++" utility and it's reporting the stats for
> that, but how it works is that "c++" (and "gcc", "g++", et al) is just a
> driver, that parses the command line arguments and shells out to the actual
> compiler ("cc1plus"), assembler and linker to get them to do all the work.
>
> If you add "-v --save-temps" to the c++ invocation, it'll show you the
> separate command lines it executes for the subprograms; the first invocation
> will be of cc1plus, using the -E flag to generate the preprocessed source into
> a .ii file, it's the second invocation you want, the one which uses the
> "-fpreprocessed" flag and names the .ii file as input, which is the one that
> actually then compiles the pre-processed source into assembly. For fuller
> explanation, see the GCC wiki:
>
> http://gcc.gnu.org/wiki/DebuggingGCC
>
> cheers,
> DaveK
>
>
>
>
o.k. after some reading and some experimenting,
I finally got firefox to compile all the way without a
memory leak..
here's what I found:
using the default .mozconfig I see during compiling there
is a -O3 in the CFLAGS(mozilla default optimization)
When using my optimization I use -O2
by mistake I accidentally forgot to add the -O2 while
tweaking my .mozconfig for a right CFLAGS to use for
debugging, in doing so firefox compiled all the way.

So using a custom CFLAGS in mozconfig and leaving out
-O* works..

Any ideas what might be happening here?

Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Justin P. Mattock · External Since: Jan 23, 2009 Posts: 29

Dave Korn wrote:
> Justin Mattock wrote:
>
>
>
>> ==1830== Command: c++ -o jsxml.o -c -DOSTYPE="Linux2.6" -DOSARCH=Linux
>>
>
> Ah, you're running it on the "c++" utility and it's reporting the stats for
> that, but how it works is that "c++" (and "gcc", "g++", et al) is just a
> driver, that parses the command line arguments and shells out to the actual
> compiler ("cc1plus"), assembler and linker to get them to do all the work.
>
> If you add "-v --save-temps" to the c++ invocation, it'll show you the
> separate command lines it executes for the subprograms; the first invocation
> will be of cc1plus, using the -E flag to generate the preprocessed source into
> a .ii file, it's the second invocation you want, the one which uses the
> "-fpreprocessed" flag and names the .ii file as input, which is the one that
> actually then compiles the pre-processed source into assembly. For fuller
> explanation, see the GCC wiki:
>
> http://gcc.gnu.org/wiki/DebuggingGCC
>
> cheers,
> DaveK
>
>
>
>
I didn't think at the time, but when compiling
4.5.*(snapshot) I added the pure64 patch to
gcc, everything seemed to go fine, but maybe
there needs to be more to it.

would this might cause an issue(memory leak or something)
like what I was receiving even though there's a symlink
to lib64?

BTW: I just cleared the deck and started fresh,
compiled gcc+pure64 patch(snapshot) into a single directory,
and then looked at ldd /usr/bin/gcc*
I see it pointing to /lib64(looking on my other system with
4.4.1 it points to /lib)

Justin P. Mattock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/