[PATCH 1/3] sched: Enable wake balancing for the SMT/HT do..

Mike Galbraith · External Since: May 26, 2006 Posts: 368

On Tue, 2009-10-27 at 15:35 +0100, Mike Galbraith wrote:
> On Mon, 2009-10-26 at 02:53 +0100, Peter Zijlstra wrote:
> > On Sun, 2009-10-25 at 23:04 +0100, Mike Galbraith wrote:
> > > if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
> > > - cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
> > > + (level == SD_LV_SIBLING || level == SD_LV_MC)) {
> >
> > quick comment without actually having looked at the patch, we should
> > really get rid of sd->level and encode properties of the sched domains
> > in sd->flags.
>
> I used SD_PREFER_SIBLING in the below. Did I break anything?

Um, other than taskset.

> + if (candidate == -1 || candidate == cpu) {
> + for_each_cpu(i, sched_domain_span(tmp)) {

if (!cpumask_test_cpu(i, &p->cpus_allowed))
continue;

(i think i'm going to need that domain flag in a few minutes anyway)

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Mike Galbraith · External Since: May 26, 2006 Posts: 368

On Wed, 2009-10-28 at 08:25 +0100, Mike Galbraith wrote:

> (i think i'm going to need that domain flag in a few minutes anyway)

Dirty trick didn't work out.

As long as I let wake_affine() decide whether an affine wakeup should
succeed based on the original cpu and then take the switched out cpu
instead, it worked ok. Try to be cleaner, and it turned to crud.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Mike Galbraith · External Since: May 26, 2006 Posts: 368

On Wed, 2009-11-04 at 21:37 +0100, Mike Galbraith wrote:
> On Wed, 2009-11-04 at 19:33 +0000, tip-bot for Mike Galbraith wrote:
> > Commit-ID: a1f84a3ab8e002159498814eaa7e48c33752b04b
> > Gitweb: http://git.kernel.org/tip/a1f84a3ab8e002159498814eaa7e48c33752b04b
> > Author: Mike Galbraith <efault.TakeThisOut@gmx.de>
> > AuthorDate: Tue, 27 Oct 2009 15:35:38 +0100
> > Committer: Ingo Molnar <mingo.TakeThisOut@elte.hu>
> > CommitDate: Wed, 4 Nov 2009 18:46:22 +0100
> >
> > sched: Check for an idle shared cache in select_task_rq_fair()
> >
> > When waking affine, check for an idle shared cache, and if
> > found, wake to that CPU/sibling instead of the waker's CPU.
>
> This one still wants some work. It's not playing well with 1b9508f.

Still does want some work (on the tail), but those numbers were from a
failed experiment kernel, not d995f1d. Below are the correct numbers.

tip v2.6.32-rc6-1731-gc5bb4b1

mysql+oltp
clients 1 2 4 8 16 32 64 128 256
tip 10447.14 19734.58 36038.18 35776.85 34662.76 33682.30 32256.22 28770.99 25323.23
10462.61 19580.14 36050.48 35942.63 35054.84 33988.40 32423.89 29259.65 25892.24
10501.02 19231.27 36007.03 35985.32 35060.79 33945.47 32400.42 29140.84 25716.16
tip avg 10470.25 19515.33 36031.89 35901.60 34926.13 33872.05 32360.17 29057.16 25643.87

tip v2.6.32-rc6-1731-gc5bb4b1 + 1b9508f (Rate-limit newidle)
tip+ 10095.64 19989.67 36449.85 35923.98 35024.24 34026.61 32411.75 28965.77 27287.96
10764.98 19864.03 36632.78 36338.00 35610.52 34439.88 32942.57 29889.18 27426.12
10734.48 19572.63 36507.90 36236.36 35486.51 34443.86 32938.90 30136.39 27186.61
tip+ avg 10531.70 19808.77 36530.17 36166.11 35373.75 34303.45 32764.40 29663.78 27300.23
1.005 1.015 1.013 1.007 1.012 1.012 1.012 1.020 1.064

v2.6.32-rc6-1796-gd995f1d
10745.30 19684.71 36367.25 35890.25 34598.08 33672.37 32327.99 28744.09 25393.00
10549.41 19747.85 36778.96 36410.58 35531.90 34292.03 32603.66 29487.77 25351.09
10678.30 19944.74 36789.44 36403.78 35523.40 34267.10 32659.60 29359.59 25751.93
avg 10657.67 19792.43 36645.21 36234.87 35217.79 34077.16 32530.41 29197.15 25498.67
vs c5bb4b1 1.017 1.014 1.017 1.009 1.008 1.006 1.005 1.004 .994

Lost the tail gain, gained peak. Drink one, spill one, give one away.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Mike Galbraith · External Since: May 26, 2006 Posts: 368

On Thu, 2009-11-05 at 10:57 +0100, Mike Galbraith wrote:

> Erk, indeed. You didn't apply the follow-up fix, and I forgot as well.

But maybe you should just back it out until I've fiddled/tested a bit
more. Dunno.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

tip-bot for Mike Galbrait · External Since: Sep 18, 2009 Posts: 3

Commit-ID: a1f84a3ab8e002159498814eaa7e48c33752b04b
Gitweb: http://git.kernel.org/tip/a1f84a3ab8e002159498814eaa7e48c33752b04b
Author: Mike Galbraith <efault.DeleteThis@gmx.de>
AuthorDate: Tue, 27 Oct 2009 15:35:38 +0100
Committer: Ingo Molnar <mingo.DeleteThis@elte.hu>
CommitDate: Wed, 4 Nov 2009 18:46:22 +0100

sched: Check for an idle shared cache in select_task_rq_fair()

When waking affine, check for an idle shared cache, and if
found, wake to that CPU/sibling instead of the waker's CPU.

This improves pgsql+oltp ramp up by roughly 8%. Possibly more
for other loads, depending on overlap. The trade-off is a
roughly 1% peak downturn if tasks are truly synchronous.

Signed-off-by: Mike Galbraith <efault.DeleteThis@gmx.de>
Cc: Arjan van de Ven <arjan.DeleteThis@infradead.org>
Cc: Peter Zijlstra <peterz.DeleteThis@infradead.org>
Cc: <stable.DeleteThis@kernel.org>
LKML-Reference: <1256654138.17752.7.camel.DeleteThis@marge.simson.net>
Signed-off-by: Ingo Molnar <mingo.DeleteThis@elte.hu>
---
kernel/sched_fair.c | 33 +++++++++++++++++++++++++++++----
1 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index 4e777b4..da87385 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -1372,11 +1372,36 @@ static int select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flag
want_sd = 0;
}

- if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
- cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
+ if (want_affine && (tmp->flags & SD_WAKE_AFFINE)) {
+ int candidate = -1, i;

- affine_sd = tmp;
- want_affine = 0;
+ if (cpumask_test_cpu(prev_cpu, sched_domain_span(tmp)))
+ candidate = cpu;
+
+ /*
+ * Check for an idle shared cache.
+ */
+ if (tmp->flags & SD_PREFER_SIBLING) {
+ if (candidate == cpu) {
+ if (!cpu_rq(prev_cpu)->cfs.nr_running)
+ candidate = prev_cpu;
+ }
+
+ if (candidate == -1 || candidate == cpu) {
+ for_each_cpu(i, sched_domain_span(tmp)) {
+ if (!cpu_rq(i)->cfs.nr_running) {
+ candidate = i;
+ break;
+ }
+ }
+ }
+ }
+
+ if (candidate >= 0) {
+ affine_sd = tmp;
+ want_affine = 0;
+ cpu = candidate;
+ }
}

if (!want_sd && !want_affine)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.DeleteThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Mike Galbraith · External Since: May 26, 2006 Posts: 368

On Thu, 2009-11-05 at 10:30 +0100, Ingo Molnar wrote:
> * tip-bot for Mike Galbraith <efault.RemoveThis@gmx.de> wrote:
>
> > Commit-ID: a1f84a3ab8e002159498814eaa7e48c33752b04b
> > Gitweb: http://git.kernel.org/tip/a1f84a3ab8e002159498814eaa7e48c33752b04b
> > Author: Mike Galbraith <efault.RemoveThis@gmx.de>
> > AuthorDate: Tue, 27 Oct 2009 15:35:38 +0100
> > Committer: Ingo Molnar <mingo.RemoveThis@elte.hu>
> > CommitDate: Wed, 4 Nov 2009 18:46:22 +0100
> >
> > sched: Check for an idle shared cache in select_task_rq_fair()
>
> -tip testing found that this causes problems:
>
> [ 26.804000] BUG: using smp_processor_id() in preemptible [00000000] code: events/1/10
> [ 26.808000] caller is vmstat_update+0x26/0x70
> [ 26.812000] Pid: 10, comm: events/1 Not tainted 2.6.32-rc5 #6887
> [ 26.816000] Call Trace:
> [ 26.820000] [<c1924a24>] ? printk+0x28/0x3c
> [ 26.824000] [<c13258a0>] debug_smp_processor_id+0xf0/0x110
> [ 26.824000] mount used greatest stack depth: 1464 bytes left
> [ 26.828000] [<c111d086>] vmstat_update+0x26/0x70
> [ 26.832000] [<c1086418>] worker_thread+0x188/0x310
> [ 26.836000] [<c10863b7>] ? worker_thread+0x127/0x310
> [ 26.840000] [<c108d310>] ? autoremove_wake_function+0x0/0x60
> [ 26.844000] [<c1086290>] ? worker_thread+0x0/0x310
> [ 26.848000] [<c108cf0c>] kthread+0x7c/0x90
> [ 26.852000] [<c108ce90>] ? kthread+0x0/0x90
> [ 26.856000] [<c100c0a7>] kernel_thread_helper+0x7/0x10
> [ 26.860000] BUG: using smp_processor_id() in preemptible [00000000] code: events/1/10
> [ 26.864000] caller is vmstat_update+0x3c/0x70
>
> oh ... doesnt it break cpus_allowed?

Erk, indeed. You didn't apply the follow-up fix, and I forgot as well.

sched: select_task_rq_fair():: add missing cpu allowed check in commit a1f84a3.

Signed-off-by: Mike Galbraith <efault.RemoveThis@gmx.de>
Cc: Ingo Molnar <mingo.RemoveThis@elte.hu>
Cc: Peter Zijlstra <a.p.zijlstra.RemoveThis@chello.nl>
LKML-Reference: <new-submission>

diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index 32f06ed..5488a5d 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -1415,6 +1415,8 @@ static int select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flag

if (candidate == -1 || candidate == cpu) {
for_each_cpu(i, sched_domain_span(tmp)) {
+ if (!cpumask_test_cpu(i, &p->cpus_allowed))
+ continue;
if (!cpu_rq(i)->cfs.nr_running) {
candidate = i;
break;

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Mike Galbraith · External Since: May 26, 2006 Posts: 368

On Wed, 2009-11-04 at 19:33 +0000, tip-bot for Mike Galbraith wrote:
> Commit-ID: a1f84a3ab8e002159498814eaa7e48c33752b04b
> Gitweb: http://git.kernel.org/tip/a1f84a3ab8e002159498814eaa7e48c33752b04b
> Author: Mike Galbraith <efault.TakeThisOut@gmx.de>
> AuthorDate: Tue, 27 Oct 2009 15:35:38 +0100
> Committer: Ingo Molnar <mingo.TakeThisOut@elte.hu>
> CommitDate: Wed, 4 Nov 2009 18:46:22 +0100
>
> sched: Check for an idle shared cache in select_task_rq_fair()
>
> When waking affine, check for an idle shared cache, and if
> found, wake to that CPU/sibling instead of the waker's CPU.

This one still wants some work. It's not playing well with 1b9508f.

tip v2.6.32-rc6-1731-gc5bb4b1

mysql+oltp
clients 1 2 4 8 16 32 64 128 256
tip 10447.14 19734.58 36038.18 35776.85 34662.76 33682.30 32256.22 28770.99 25323.23
10462.61 19580.14 36050.48 35942.63 35054.84 33988.40 32423.89 29259.65 25892.24
10501.02 19231.27 36007.03 35985.32 35060.79 33945.47 32400.42 29140.84 25716.16
tip avg 10470.25 19515.33 36031.89 35901.60 34926.13 33872.05 32360.17 29057.16 25643.87

tip v2.6.32-rc6-1731-gc5bb4b1 + 1b9508f (Rate-limit newidle)
tip+ 10095.64 19989.67 36449.85 35923.98 35024.24 34026.61 32411.75 28965.77 27287.96
10764.98 19864.03 36632.78 36338.00 35610.52 34439.88 32942.57 29889.18 27426.12
10734.48 19572.63 36507.90 36236.36 35486.51 34443.86 32938.90 30136.39 27186.61
tip+ avg 10531.70 19808.77 36530.17 36166.11 35373.75 34303.45 32764.40 29663.78 27300.23
1.005 1.015 1.013 1.007 1.012 1.012 1.012 1.020 1.064

v2.6.32-rc6-1796-gd995f1d (a1f84a3 not playing well with 1b9508f)
10578.44 19358.96 36184.70 35549.19 34625.40 33658.72 31947.51 27504.11 20400.97
10610.76 19748.84 36401.08 36021.75 34830.67 33566.52 31989.97 27622.25 21348.00
10568.87 19600.60 36190.48 35959.96 34805.45 33641.46 32005.30 27591.48 22509.01

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar · External Since: May 15, 2006 Posts: 3111

* tip-bot for Mike Galbraith <efault RemoveThis @gmx.de> wrote:

> Commit-ID: a1f84a3ab8e002159498814eaa7e48c33752b04b
> Gitweb: http://git.kernel.org/tip/a1f84a3ab8e002159498814eaa7e48c33752b04b
> Author: Mike Galbraith <efault RemoveThis @gmx.de>
> AuthorDate: Tue, 27 Oct 2009 15:35:38 +0100
> Committer: Ingo Molnar <mingo RemoveThis @elte.hu>
> CommitDate: Wed, 4 Nov 2009 18:46:22 +0100
>
> sched: Check for an idle shared cache in select_task_rq_fair()

-tip testing found that this causes problems:

[ 26.804000] BUG: using smp_processor_id() in preemptible [00000000] code: events/1/10
[ 26.808000] caller is vmstat_update+0x26/0x70
[ 26.812000] Pid: 10, comm: events/1 Not tainted 2.6.32-rc5 #6887
[ 26.816000] Call Trace:
[ 26.820000] [<c1924a24>] ? printk+0x28/0x3c
[ 26.824000] [<c13258a0>] debug_smp_processor_id+0xf0/0x110
[ 26.824000] mount used greatest stack depth: 1464 bytes left
[ 26.828000] [<c111d086>] vmstat_update+0x26/0x70
[ 26.832000] [<c1086418>] worker_thread+0x188/0x310
[ 26.836000] [<c10863b7>] ? worker_thread+0x127/0x310
[ 26.840000] [<c108d310>] ? autoremove_wake_function+0x0/0x60
[ 26.844000] [<c1086290>] ? worker_thread+0x0/0x310
[ 26.848000] [<c108cf0c>] kthread+0x7c/0x90
[ 26.852000] [<c108ce90>] ? kthread+0x0/0x90
[ 26.856000] [<c100c0a7>] kernel_thread_helper+0x7/0x10
[ 26.860000] BUG: using smp_processor_id() in preemptible [00000000] code: events/1/10
[ 26.864000] caller is vmstat_update+0x3c/0x70

oh ... doesnt it break cpus_allowed?

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo RemoveThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

tip-bot for Mike Galbrait · External Since: Sep 18, 2009 Posts: 3

Commit-ID: fd210738f6601d0fb462df9a2fe5a41896ff6a8f
Gitweb: http://git.kernel.org/tip/fd210738f6601d0fb462df9a2fe5a41896ff6a8f
Author: Mike Galbraith <efault.TakeThisOut@gmx.de>
AuthorDate: Thu, 5 Nov 2009 10:57:46 +0100
Committer: Ingo Molnar <mingo.TakeThisOut@elte.hu>
CommitDate: Thu, 5 Nov 2009 11:01:39 +0100

sched: Fix affinity logic in select_task_rq_fair()

Ingo Molnar reported:

[ 26.804000] BUG: using smp_processor_id() in preemptible [00000000] code: events/1/10
[ 26.808000] caller is vmstat_update+0x26/0x70
[ 26.812000] Pid: 10, comm: events/1 Not tainted 2.6.32-rc5 #6887
[ 26.816000] Call Trace:
[ 26.820000] [<c1924a24>] ? printk+0x28/0x3c
[ 26.824000] [<c13258a0>] debug_smp_processor_id+0xf0/0x110
[ 26.824000] mount used greatest stack depth: 1464 bytes left
[ 26.828000] [<c111d086>] vmstat_update+0x26/0x70
[ 26.832000] [<c1086418>] worker_thread+0x188/0x310
[ 26.836000] [<c10863b7>] ? worker_thread+0x127/0x310
[ 26.840000] [<c108d310>] ? autoremove_wake_function+0x0/0x60
[ 26.844000] [<c1086290>] ? worker_thread+0x0/0x310
[ 26.848000] [<c108cf0c>] kthread+0x7c/0x90
[ 26.852000] [<c108ce90>] ? kthread+0x0/0x90
[ 26.856000] [<c100c0a7>] kernel_thread_helper+0x7/0x10
[ 26.860000] BUG: using smp_processor_id() in preemptible [00000000] code: events/1/10
[ 26.864000] caller is vmstat_update+0x3c/0x70

Because this commit:

a1f84a3: sched: Check for an idle shared cache in select_task_rq_fair()

broke ->cpus_allowed.

Signed-off-by: Mike Galbraith <efault.TakeThisOut@gmx.de>
Cc: Peter Zijlstra <a.p.zijlstra.TakeThisOut@chello.nl>
Cc: arjan.TakeThisOut@infradead.org
Cc: <stable.TakeThisOut@kernel.org>
LKML-Reference: <1257415066.12867.1.camel.TakeThisOut@marge.simson.net>
Signed-off-by: Ingo Molnar <mingo.TakeThisOut@elte.hu>
---
kernel/sched_fair.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index da87385..e4d4483 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -1389,6 +1389,8 @@ static int select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flag

if (candidate == -1 || candidate == cpu) {
for_each_cpu(i, sched_domain_span(tmp)) {
+ if (!cpumask_test_cpu(i, &p->cpus_allowed))
+ continue;
if (!cpu_rq(i)->cfs.nr_running) {
candidate = i;
break;
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.TakeThisOut@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Peter Zijlstra · External Since: Jun 06, 2007 Posts: 205

On Sun, 2009-10-25 at 22:36 -0700, Arjan van de Ven wrote:
>
> if you're a multicore domain you better have a shared cache.. that's
> what it should mean. If it does not we should fix that.

Fully agreed.. I've been dying to do the below for ages Smile

---
arch/x86/kernel/smpboot.c | 10 +---------
1 files changed, 1 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 213a7a3..297b307 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -433,15 +433,7 @@ void __cpuinit set_cpu_sibling_map(int cpu)
const struct cpumask *cpu_coregroup_mask(int cpu)
{
struct cpuinfo_x86 *c = &cpu_data(cpu);
- /*
- * For perf, we return last level cache shared map.
- * And for power savings, we return cpu_core_map
- */
- if ((sched_mc_power_savings || sched_smt_power_savings) &&
- !(cpu_has(c, X86_FEATURE_AMD_DCM)))
- return cpu_core_mask(cpu);
- else
- return c->llc_shared_map;
+ return c->llc_shared_map;
}

static void impress_friends(void)

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo DeleteThis @vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Arjan van de Ven · External Since: May 15, 2006 Posts: 901

On Tue, 10 Nov 2009 22:59:29 +0100
Peter Zijlstra <peterz.RemoveThis@infradead.org> wrote:

> On Sun, 2009-10-25 at 22:36 -0700, Arjan van de Ven wrote:
> >
> > if you're a multicore domain you better have a shared cache.. that's
> > what it should mean. If it does not we should fix that.
>
> Fully agreed.. I've been dying to do the below for ages Smile

>

completely sane.

Acked-by: Arjan van de Ven <arjan.RemoveThis@linux.intel.com>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo.RemoveThis@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/