Debian powerpc binary compatibility?

Sergei Poselenov · External Since: Sep 17, 2010 Posts: 7

Hello all,

Could you please answer some questions, regarding the use of the
Debian powerpc distro for embedded PowerPC system?

- What exact target CPU it is built for? I see here
http://www.debian.org/ports/powerpc/inst/install the list of
supported machines, but how can I understand will the Debian binaries
work on my board with the specific CPU?
Is it possible, knowing only the specific CPU type (440, 603, 8540,
8560, etc) to say will it run Debian PowerPC Distro or not?
(I understand that there should be a kernel supporting the specific
board; I'm rather interested in the instructions set used by GCC to
build the binary images for Debian powerpc distro. In short - what
"-mcpu=" GCC option is used for build?)

- We are considering to rebuild the Debian distribution, to
tune the packages for the specific CPU type (enable hardware FP
support, etc).
I now the Debian powerpc distro is built natively. Are there any
details of the build process available, so we could use it in our
build system?

Thanks for any help.

Regards,
Sergei

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org
Archive: http://lists.debian.org/20100917134819.08536b8a@emcraft.com

Sebastian Andrzej Siewior · External Since: Jan 20, 2009 Posts: 59

* Sergei Poselenov | 2010-09-17 13:48:19 [+0400]:

>Could you please answer some questions, regarding the use of the
>Debian powerpc distro for embedded PowerPC system?
>
>- What exact target CPU it is built for? I see here
> http://www.debian.org/ports/powerpc/inst/install the list of
> supported machines, but how can I understand will the Debian binaries
> work on my board with the specific CPU?
> Is it possible, knowing only the specific CPU type (440, 603, 8540,
> 8560, etc) to say will it run Debian PowerPC Distro or not?
- 603 are fine.
- I have no clue whether 440 has a FPU or not. If it don't you need
either to enable enable floating emulation in kernel or a softfloat
port.
- 8540 and 8560 are e500v1 core based cpus. They use the SPE unit for
single precision floating point operation. This is not compatible with
the "classic" floating unit in other powerpc like the 603. You may use
the kernel emulation to work around the problem but they remain a few
opcodes which trigger an invalid opcode.

> (I understand that there should be a kernel supporting the specific
> board; I'm rather interested in the instructions set used by GCC to
> build the binary images for Debian powerpc distro. In short - what
> "-mcpu=" GCC option is used for build?)

generic so you can run it on any powerpc. Floating point could be a
problem which can solved by kernel emulation which will make FPU
intesive operation slow. On 85xx lwsync will be another problem.

>- We are considering to rebuild the Debian distribution, to
> tune the packages for the specific CPU type (enable hardware FP
> support, etc).
> I now the Debian powerpc distro is built natively. Are there any
> details of the build process available, so we could use it in our
> build system?

For powerpcspe [0] we use powerpc-linux-gnuspe as gcc tripplet which
enables the correct CPU and floating point for us. There is no more
magic behind it.
If you want to rebuild the complete Debian with -mcpu=603 then the
easiest thing would be to put a wrapper around gcc which passes the
correct CFLAGS _or_ make sure that the default cpu is 603.

[0] http://wiki.debian.org/PowerPCSPEPort

>Thanks for any help.
>
>Regards,
>Sergei

Sebastian

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org
Archive: http://lists.debian.org/20100917181929.GA5327@Chamillionaire.breakpoint.cc

Sergei Poselenov · External Since: Sep 17, 2010 Posts: 7

Hello Sebastian,

Thanks for reply! Please see my comments below.

On Fri, 17 Sep 2010 20:19:29 +0200
Sebastian Andrzej Siewior wrote:

> * Sergei Poselenov | 2010-09-17 13:48:19 [+0400]:
>
> >Could you please answer some questions, regarding the use of the
> >Debian powerpc distro for embedded PowerPC system?
> >
> >- What exact target CPU it is built for? I see here
> > http://www.debian.org/ports/powerpc/inst/install the list of
> > supported machines, but how can I understand will the Debian
> > binaries work on my board with the specific CPU?
> > Is it possible, knowing only the specific CPU type (440, 603, 8540,
> > 8560, etc) to say will it run Debian PowerPC Distro or not?
> - 603 are fine.
> - I have no clue whether 440 has a FPU or not. If it don't you need
> either to enable enable floating emulation in kernel or a softfloat
> port.

Thanks, but I'm looking for specific information - what instruction set
was used by gcc to build Debian binaries.

It seems to me this information is not readily available... I can only
guess of gcc configuration - just defaults (-mcpu=powepc -msoft-float).

> - 8540 and 8560 are e500v1 core based cpus. They use the SPE unit for
> single precision floating point operation. This is not compatible
> with the "classic" floating unit in other powerpc like the 603. You
> may use the kernel emulation to work around the problem but they
> remain a few opcodes which trigger an invalid opcode.
>
> > (I understand that there should be a kernel supporting the specific
> > board; I'm rather interested in the instructions set used by GCC to
> > build the binary images for Debian powerpc distro. In short - what
> > "-mcpu=" GCC option is used for build?)
>
> generic so you can run it on any powerpc. Floating point could be a
> problem which can solved by kernel emulation which will make FPU
> intesive operation slow. On 85xx lwsync will be another problem.
>

Yes, I'm aware of an attempt to run Debian on e500v2 - it throws
"Illegal instruction" on "lwsync". I'm sorry I don't have enough
details - what Debian Distro they tried and what kernel the target
has been run.

However, speaking about the "lwsync" problem on e500 - I've got an
impression that the problem should be solved already -
http://gcc.gnu.org/ml/gcc-patches/2006-11/msg01238.html

Do you know - isn't the fix in the mainline gcc?

And while we at this: on the [0], the current status of the project is
"stalled" because of GCC bug in e500 floating point insns. The bug is
reported against 4.4.4.

Have you tried other Debian GCC version - 4.3? Is the bug there, too?

> >- We are considering to rebuild the Debian distribution, to
> > tune the packages for the specific CPU type (enable hardware FP
> > support, etc).
> > I now the Debian powerpc distro is built natively. Are there any
> > details of the build process available, so we could use it in our
> > build system?
>
> For powerpcspe [0] we use powerpc-linux-gnuspe as gcc tripplet which
> enables the correct CPU and floating point for us. There is no more
> magic behind it.
> If you want to rebuild the complete Debian with -mcpu=603 then the
> easiest thing would be to put a wrapper around gcc which passes the
> correct CFLAGS _or_ make sure that the default cpu is 603.
>

Yes, I understand this. I guess, for your powerpcspe port you are
rebuilding Debian packages on the system running the same Debian,
right? (What Debian are running on your e500, btw? How your overcame
the "lwsync" problem?)

Probably, I was not quite clear in my question - we are considering
the case that there will be no working Debian distribution for
particular target (like e500 "lwsync" problem I mentioned above), so we
will have to use some other distribution for native build system, with
different toolchain version (I'mean the versions of gcc, binutils and
glibc, which could be different of those in Debian toolchain). How
could I rebuild Debian on such system?

Apparently, I need first to build the Debian toolchain, and then make
sure the binaries are linked against the new-built Glibc, not the build
host one. Also, because of the package dependencies, the order of
package's build is also important.

So, the question is - how to build Debian "from scratch"?

Again, thanks for reply.

Regards,
Sergei

> [0] http://wiki.debian.org/PowerPCSPEPort
>
> >Thanks for any help.
> >
> >Regards,
> >Sergei
>
> Sebastian

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org
Archive: http://lists.debian.org/20100918114401.0b75b9a7@emcraft.com

Sergei Poselenov · External Since: Sep 17, 2010 Posts: 7

Hello again, Sebastian,

A question regarding the Debian installation to e500 system:
As I understand, standard Debian powerpc won't work on e500 because of
"lwsync" problem, right?

Here are the instructions on how to boot you e500 machine:
http://wiki.debian.org/PowerPCSPEPort#Installingtheport

"...
edit etc/apt/sources.list and enter a mirror from here which is close to you.
the password for root is root
...."

Should I proceed with apt-get after boot?
Will it load the complete Debian, with toolchain?

Thanks in advance,
Sergei

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org
Archive: http://lists.debian.org/20100918142740.4bb0a1fa@emcraft.com

Sebastian Andrzej Siewior · External Since: Jan 20, 2009 Posts: 59

* Sergei Poselenov | 2010-09-18 14:27:40 [+0400]:

>Hello again, Sebastian,
Hello,

>A question regarding the Debian installation to e500 system:
>As I understand, standard Debian powerpc won't work on e500 because of
>"lwsync" problem, right?
Right. Every C++ program will throw SIGILL because libstdc++ uses this
opcode in one of its init functions.
I've seen core documentation which says that they have a bit in the HID1
register of the CPU to enable lwsync and don't threat it as an invalid
opcode. However I haven't find a CPU which implements this bit i.e. the
CPU's reference manual treats this bit as reserved.

>Here are the instructions on how to boot you e500 machine:
>http://wiki.debian.org/PowerPCSPEPort#Installingtheport
>
>"...
>edit etc/apt/sources.list and enter a mirror from here which is close to you.
>the password for root is root
>..."
>
>Should I proceed with apt-get after boot?
Yes. You can boot into the minimal environment and install packages you
need with apt-get.

>Will it load the complete Debian, with toolchain?
you can execute
apt-get install build-essential

and this will install gcc & binutils.

>Thanks in advance,
>Sergei

Sebastian

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org
Archive: http://lists.debian.org/20100919204327.GB21972@Chamillionaire.breakpoint.cc

Sergei Poselenov · External Since: Sep 17, 2010 Posts: 7

Hello Sebastian,

Should the bug 44606 trigger on a e500v1 CPU?

I can check it with gcc 4.2.2 on a MPC8560 system.

Regards,
Sergei

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org
Archive: http://lists.debian.org/20100921145422.21377841@emcraft.com

Sebastian Andrzej Siewior · External Since: Jan 20, 2009 Posts: 59

* Sergei Poselenov | 2010-09-21 14:54:22 [+0400]:

>Hello Sebastian,
Hi Sergei,

>Should the bug 44606 trigger on a e500v1 CPU?
>
>I can check it with gcc 4.2.2 on a MPC8560 system.
SPE support got merged in gcc 4.3. An e500v1 CPU does not have support
for the type "double" so you shouldn't be able run the binary if you
exlicitly enable using it.
On the other hand, nobody is using 4.2 these days so it is not an option
to switch to an older gcc.

>Regards,
>Sergei

Sebastian

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org
Archive: http://lists.debian.org/20100921192326.GB7569@Chamillionaire.breakpoint.cc

Sebastian Andrzej Siewior · External Since: Jan 20, 2009 Posts: 59

* Sergei Poselenov | 2010-09-21 14:09:24 [+0400]:

>Hello Sebastian,
Hi Sergei,

>BTW, do you know - why the separate target for e500? Is the instruction
>set for e500 so fundamentally different it needs a special configuration
>for GCC? Why GCC cannot switch codegeneration at run-time, conditioning
>on "-mcpu=e500"?

Because the ABI is different. All you need to the same output as with
the tripplet you have to specify
-mcpu=mpc8540 -misel -mfloat-gprs=double
So with the tripplet you set all this flags as default (unless I forgot
a switch). -mfloat-gprs changes the ABI from hardfloat to softfloat.
That means you can't mix the binaries, they are not longer compatible.

>Having the separate target looks inconvenient for me - you have to
>"push" the target to all package, so they wouldn't fail on "configure
>--target=powerpc-linux-gnuspe".

No, you don't have. The dpkg-architecure command on powerpcspe port
returns:

|# dpkg-architecture |grep TYPE
|DEB_BUILD_GNU_TYPE=powerpc-linux-gnuspe
|DEB_HOST_GNU_TYPE=powerpc-linux-gnuspe

That means all packages that pass --target usually go for
DEB_HOST_GNU_TYPE. If you don't specify anything i.e. a simply
../configure then it is also correct since it goes for gcc which has
allready the correct default.

>> Yes I can reproduce it on 4.3.
>>
>
>This is bad. Does this mean the modern GCC cannot be used for e500
>currently?

Some testcases pass, some don't. So yes you may get invalid code. To
workaround you can disable scheduling in gcc as pointed out in the PR.
You can probably also avoid the bug if you use -mfloat-gprs=single.

>Mmm, do you mean that you are running the Debian rebuilt with gnuspe
>GCC and [0] binutils fix?
Yes that is the case.

>Have you considered to fix the "lwsync" problem in kernel? I mean
>to catch the Illegal Instruction exception and switch the faulted lwsync
>to sync?
I've been thinking about it but didn't go for it. The problem with that
solution is that you are required to emulate the floating point code in
kernel leaving the "floating unit" untouched. I posted some numbers
here [0] and there [1] and it didn't look good. So the result is not
worth it.

[0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=520877#48
[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=575158#57

>
>Regards,
>Sergei

Sebastian

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org
Archive: http://lists.debian.org/20100921191847.GA7569@Chamillionaire.breakpoint.cc

Sergei Poselenov · External Since: Sep 17, 2010 Posts: 7

Hi Sebastian,

On Tue, 21 Sep 2010 21:23:26 +0200
Sebastian Andrzej Siewior wrote:

> * Sergei Poselenov | 2010-09-21 14:54:22 [+0400]:
>
> >Hello Sebastian,
> Hi Sergei,
>
> >Should the bug 44606 trigger on a e500v1 CPU?
> >
> >I can check it with gcc 4.2.2 on a MPC8560 system.
> SPE support got merged in gcc 4.3. An e500v1 CPU does not have support
> for the type "double" so you shouldn't be able run the binary if you
> exlicitly enable using it.
> On the other hand, nobody is using 4.2 these days so it is not an
> option to switch to an older gcc.
>

Understand. However, FYI: testcase built with GCC 4.2 compiler passed
OK:
-bash-3.2# ./tc0
..26 3264.000000
..28 3520.000000
..30 3776.000000
..32 4032.000000
..34 4288.000000
-bash-3.2# ./tc2
..26 3264.000000
..28 3520.000000
..30 3776.000000
..32 4032.000000
..34 4288.000000
-bash-3.2#

(I verified the binary image has the SPE instructions:

10000590 <MinifyImage>:
10000590: 94 21 ff 70 stwu r1,-144(r1)
10000594: 7c 08 02 a6 mflr r0
10000598: 13 e1 83 21 vmhraddshs v31,v1,v16,v12
1000059c: 90 01 00 94 stw r0,148(r1)
100005a0: 7c 3f 0b 78 mr r31,r1
100005a4: 90 7f 00 68 stw r3,104(r31)
....)

Also, just tried the CodeSourcery GCC 4.4.1 - the bug is here.

Regards,
Sergei

> >Regards,
> >Sergei
>
> Sebastian

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org
Archive: http://lists.debian.org/20100922143324.0cb9bda4@emcraft.com

Sergei Poselenov · External Since: Sep 17, 2010 Posts: 7

Hi Sebastian,

I'm sorry I'm a bit slow in understanding - could you please answer my
further questions?

On Tue, 21 Sep 2010 21:18:47 +0200
Sebastian Andrzej Siewior wrote:

> * Sergei Poselenov | 2010-09-21 14:09:24 [+0400]:
>
> >Hello Sebastian,
> Hi Sergei,
>
> >BTW, do you know - why the separate target for e500? Is the
> >instruction set for e500 so fundamentally different it needs a
> >special configuration for GCC? Why GCC cannot switch codegeneration
> >at run-time, conditioning on "-mcpu=e500"?
>
> Because the ABI is different. All you need to the same output as with
> the tripplet you have to specify
> -mcpu=mpc8540 -misel -mfloat-gprs=double
> So with the tripplet you set all this flags as default (unless I
> forgot a switch). -mfloat-gprs changes the ABI from hardfloat to

Probably also -mabi=spe?

> softfloat. That means you can't mix the binaries, they are not longer
> compatible.
>

I hope I understand now.
You mean I can't build an application in the
following way:

powerpc-linux-gcc -o t.o -c t.c -mcpu=mpc8540 -misel -mfloat-gprs=double
-mabi=spe
powerpc-linux-gcc -o t t.o

In the first step, I built an object with ABI=spe, on the second - I'm
linking the object with libc.so, which is built with the "default"
powerpc ABI.
I can't do this, the objects (.o and libc.so and startup modules
(crt..)) are incompatible, right?

> >Having the separate target looks inconvenient for me - you have to
> >"push" the target to all package, so they wouldn't fail on "configure
> >--target=powerpc-linux-gnuspe".
>
> No, you don't have. The dpkg-architecure command on powerpcspe port
> returns:
>
> |# dpkg-architecture |grep TYPE
> |DEB_BUILD_GNU_TYPE=powerpc-linux-gnuspe
> |DEB_HOST_GNU_TYPE=powerpc-linux-gnuspe
>
> That means all packages that pass --target usually go for
> DEB_HOST_GNU_TYPE. If you don't specify anything i.e. a simply
> ./configure then it is also correct since it goes for gcc which has
> allready the correct default.
>

Not sure I understood. I mean the bug list for the powerpcspe
http://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=powerpcspe;users=debi...powerpc
port is full of patches which essentially add the new powerpcspe
configuration target to a lot of packages.
I mean - if you use the traditional "powerpc-linux-gnu" target for
you port with the e500v2 GCC options enabled by one or other way - you
don't need to bother with the new "gnuspe" target support in a lot of
packages.
Why the decision to use the gnuspe target?

> >> Yes I can reproduce it on 4.3.
> >>
> >
> >This is bad. Does this mean the modern GCC cannot be used for e500
> >currently?
>
> Some testcases pass, some don't. So yes you may get invalid code. To
> workaround you can disable scheduling in gcc as pointed out in the PR.
> You can probably also avoid the bug if you use -mfloat-gprs=single.
>

Yes, we are considering to use the -mfloat-gprs=single as a workaround
for our port. Thanks.

> >Mmm, do you mean that you are running the Debian rebuilt with gnuspe
> >GCC and [0] binutils fix?
> Yes that is the case.
>
> >Have you considered to fix the "lwsync" problem in kernel? I mean
> >to catch the Illegal Instruction exception and switch the faulted
> >lwsync to sync?
> I've been thinking about it but didn't go for it. The problem with
> that solution is that you are required to emulate the floating point
> code in kernel leaving the "floating unit" untouched. I posted some
> numbers here [0] and there [1] and it didn't look good. So the result
> is not worth it.
>
> [0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=520877#48
> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=575158#57
>

Understood. Definitely, FP emulation is worse than natural computing.

Thanks a lot for your replies!

Regards,
Sergei

> >
> >Regards,
> >Sergei
>
> Sebastian

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org
Archive: http://lists.debian.org/20100922154939.43af67d9@emcraft.com

Sebastian Andrzej Siewior · External Since: Jan 20, 2009 Posts: 59

* Sergei Poselenov | 2010-09-22 15:49:39 [+0400]:

>Hi Sebastian,
Hi Sergei,

>I'm sorry I'm a bit slow in understanding - could you please answer my
>further questions?
hehe, let's see.

>> Because the ABI is different. All you need to the same output as with
>> the tripplet you have to specify
>> -mcpu=mpc8540 -misel -mfloat-gprs=double
>> So with the tripplet you set all this flags as default (unless I
>> forgot a switch). -mfloat-gprs changes the ABI from hardfloat to
>
>Probably also -mabi=spe?
Right, I missed that one.

>I hope I understand now.
>You mean I can't build an application in the
>following way:
>
>powerpc-linux-gcc -o t.o -c t.c -mcpu=mpc8540 -misel -mfloat-gprs=double
>-mabi=spe
>powerpc-linux-gcc -o t t.o
>
>In the first step, I built an object with ABI=spe, on the second - I'm
>linking the object with libc.so, which is built with the "default"
>powerpc ABI.
>I can't do this, the objects (.o and libc.so and startup modules
>(crt..)) are incompatible, right?
Right.

>Not sure I understood. I mean the bug list for the powerpcspe
>http://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=powerpcspe;users=debian-powerpcspe@breakpoint.cc
>port is full of patches which essentially add the new powerpcspe
>configuration target to a lot of packages.
>I mean - if you use the traditional "powerpc-linux-gnu" target for
>you port with the e500v2 GCC options enabled by one or other way - you
>don't need to bother with the new "gnuspe" target support in a lot of
>packages.
>Why the decision to use the gnuspe target?
Because you can't distinguish between the e500v2 binaries (with SPE
floating point) and the "normal" powerpc binaries with classic FPU and
AltiVec. The name is _<debian_name>.deb. You can't run those on the
"traditional" powerpc so have to ensure you don't mix it up. How do you
want to handle this?
Some packags, like gcc, have a different configuration and we pass
different options to configure. openjdk will be another one as soon as
openjdk for powerpc starts using llvm. It is not the majority, true. The
"add powercpspe keyword to bla" type of patch could be replaced with
"replace linux-any with $arch_list" these days. However it was not
allowed by the policy at the time we started doing it. This is allowed
since 3.9.0

>Regards,
>Sergei

Sebastian

--
To UNSUBSCRIBE, email to debian-powerpc-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org
Archive: http://lists.debian.org/20100923160127.GA26549@Chamillionaire.breakpoint.cc