Cross compiler ITP (armel)

Hector Oron · External Since: Jul 28, 2009 Posts: 6

Hello,

I would like to do a little explanation on the ITP I have filled for
{linux,binutils,eglibc,gcc-4.3,gcc-4.4,gdb}-armel.

These set of packages provide a cross toolchain for armel targets to
be built on i386 and amd64 platforms (maybe ppc could be added)

In order to avoid code duplication in the archive, this packages
build depend on -source packages.

As major technical issues, I would try to build cross compilers
with --sysroot support, but that means dpkg-cross need to be updated
for sysroot paths. For now, we might take the road we have been doing
at emdebian.org (for many years) and start changing bits towards a
nice sysrooted solution.

Please, let me know if you need farther explanations on some topics
or if you have any comment.

Kind regards,
--
Héctor Orón

--
To UNSUBSCRIBE, email to debian-devel-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Matthias Klose · External Since: Aug 28, 2004 Posts: 1198

On 02.11.2009 00:00, Ben Hutchings wrote:
> On Sun, 2009-11-01 at 23:14 +0100, Hector Oron wrote:
>> Hello,
>>
>> I would like to do a little explanation on the ITP I have filled for
>> {linux,binutils,eglibc,gcc-4.3,gcc-4.4,gdb}-armel.
>>
>> These set of packages provide a cross toolchain for armel targets to
>> be built on i386 and amd64 platforms (maybe ppc could be added)
>>
>> In order to avoid code duplication in the archive, this packages
>> build depend on -source packages.
> [...]
>
> At present, there is nothing that will ensure that the binary packages
> you build are released along with the packages containing their actual
> source. It would therefore require manual attention to ensure that we
> meet the source distribution requirements of the GPL, which the FTP team
> really hates having to do.
>
> Until the FTP team implements a means of automatically recording some
> build-dependencies and the versions actually used as additional source
> dependencies, and ensuring that these source dependencies are satisfied
> within each release, you should not use this approach.

I disagree. It's not worse than the current scheme splitting up GCC uploads into
three different source packages, forced by the release team.

Matthias

--
To UNSUBSCRIBE, email to debian-devel-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Bernd Zeimetz · External Since: Jul 17, 2006 Posts: 228

Ben Hutchings wrote:

> You can disagree all you like, but I believe that the FTP team will
> currently reject any new packages that use source code from their build-
> dependencies. It would likely be a waste of Hector's time to continue
> with this approach.

So if that is a problem - why not enhance the gcc packaging to build the
cross-compiler packages?

--
Bernd Zeimetz Debian GNU/Linux Developer
http://bzed.de http://www.debian.org
GPG Fingerprints: 06C8 C9A2 EAAD E37E 5B2C BE93 067A AD04 C93B FF79
ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F

--
To UNSUBSCRIBE, email to debian-devel-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Matthias Klose · External Since: Aug 28, 2004 Posts: 1198

On 02.11.2009 03:19, Ben Hutchings wrote:
> On Mon, 2009-11-02 at 02:34 +0100, Matthias Klose wrote:
>> On 02.11.2009 00:00, Ben Hutchings wrote:
>>> On Sun, 2009-11-01 at 23:14 +0100, Hector Oron wrote:
>>>> Hello,
>>>>
>>>> I would like to do a little explanation on the ITP I have filled for
>>>> {linux,binutils,eglibc,gcc-4.3,gcc-4.4,gdb}-armel.
>>>>
>>>> These set of packages provide a cross toolchain for armel targets to
>>>> be built on i386 and amd64 platforms (maybe ppc could be added)
>>>>
>>>> In order to avoid code duplication in the archive, this packages
>>>> build depend on -source packages.
>>> [...]
>>>
>>> At present, there is nothing that will ensure that the binary packages
>>> you build are released along with the packages containing their actual
>>> source. It would therefore require manual attention to ensure that we
>>> meet the source distribution requirements of the GPL, which the FTP team
>>> really hates having to do.
>>>
>>> Until the FTP team implements a means of automatically recording some
>>> build-dependencies and the versions actually used as additional source
>>> dependencies, and ensuring that these source dependencies are satisfied
>>> within each release, you should not use this approach.
>>
>> I disagree. It's not worse than the current scheme splitting up GCC uploads into
>> three different source packages, forced by the release team.
>
> You can disagree all you like, but I believe that the FTP team will
> currently reject any new packages that use source code from their build-
> dependencies. It would likely be a waste of Hector's time to continue
> with this approach.

You can believe all you like, but this is the approach which was communicated to
the FTP team at Debconf, and didn't find resistance.

--
To UNSUBSCRIBE, email to debian-devel-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Mike Hommey · External Since: Nov 09, 2004 Posts: 966

On Mon, Nov 02, 2009 at 12:25:16PM +0100, Bernd Zeimetz wrote:
> Ben Hutchings wrote:
>
> > You can disagree all you like, but I believe that the FTP team will
> > currently reject any new packages that use source code from their build-
> > dependencies. It would likely be a waste of Hector's time to continue
> > with this approach.
>
> So if that is a problem - why not enhance the gcc packaging to build the
> cross-compiler packages?

Combinatorial explosion ?

Mike

--
To UNSUBSCRIBE, email to debian-devel-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Philipp Kern · External Since: Jun 22, 2009 Posts: 8

On 2009-11-02, Hector Oron <hector.oron RemoveThis @gmail.com> wrote:
> 2009/11/2 Mike Hommey <mh RemoveThis @glandium.org>:
>> On Mon, Nov 02, 2009 at 12:25:16PM +0100, Bernd Zeimetz wrote:
>>> So if that is a problem - why not enhance the gcc packaging to build the
>>> cross-compiler packages?
>> Combinatorial explosion ?
> We took this approach, and we have been building this way.
> Binutils, GCC, GDB, EGLIBC have cross built in capability, but some
> tricks (relaying on dpkg-cross) must be done in order to build the
> cross toolchain and Debian autobuilders do not know how to keep with
> this. This is the reason we have been keeping the cross tools at
> emdebian.org site.
>
> Then relaying on -source packages approach was suggested by Matthias
> and not entirely rejected by Ganeff when I talked to him about it. A
> visualization of the build dependencies can be found at:
> http://www.emdebian.org/~zumbi/docs/deps.pdf
>
> There already packages in the archive build depending on -source, like
> binutils-z80, which is not much different from binutils-armel
> approach.

Of course it is a sane approach but very special care needs to be taken when
releasing to ensure GPL compliance. So what we should get is support in the
toolchain to declare against what source package the upload was built to
keep that around.

Kind regards,
Philipp Kern

--
To UNSUBSCRIBE, email to debian-devel-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Hector Oron · External Since: Jul 28, 2009 Posts: 6

Hi,

2009/11/2 Mike Hommey <mh RemoveThis @glandium.org>:
> On Mon, Nov 02, 2009 at 12:25:16PM +0100, Bernd Zeimetz wrote:
>> So if that is a problem - why not enhance the gcc packaging to build the
>> cross-compiler packages?
>
> Combinatorial explosion ?

We took this approach, and we have been building this way.
Binutils, GCC, GDB, EGLIBC have cross built in capability, but some
tricks (relaying on dpkg-cross) must be done in order to build the
cross toolchain and Debian autobuilders do not know how to keep with
this. This is the reason we have been keeping the cross tools at
emdebian.org site.

Then relaying on -source packages approach was suggested by Matthias
and not entirely rejected by Ganeff when I talked to him about it. A
visualization of the build dependencies can be found at:
http://www.emdebian.org/~zumbi/docs/deps.pdf

There already packages in the archive build depending on -source, like
binutils-z80, which is not much different from binutils-armel
approach.

--
Héctor Orón

--
To UNSUBSCRIBE, email to debian-devel-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Goswin von Brederlow · External Since: Feb 09, 2009 Posts: 90

Ben Hutchings <ben DeleteThis @decadent.org.uk> writes:

> On Sun, 2009-11-01 at 23:14 +0100, Hector Oron wrote:
>> Hello,
>>
>> I would like to do a little explanation on the ITP I have filled for
>> {linux,binutils,eglibc,gcc-4.3,gcc-4.4,gdb}-armel.
>>
>> These set of packages provide a cross toolchain for armel targets to
>> be built on i386 and amd64 platforms (maybe ppc could be added)
>>
>> In order to avoid code duplication in the archive, this packages
>> build depend on -source packages.
> [...]
>
> At present, there is nothing that will ensure that the binary packages
> you build are released along with the packages containing their actual
> source. It would therefore require manual attention to ensure that we
> meet the source distribution requirements of the GPL, which the FTP team
> really hates having to do.
>
> Until the FTP team implements a means of automatically recording some
> build-dependencies and the versions actually used as additional source
> dependencies, and ensuring that these source dependencies are satisfied
> within each release, you should not use this approach.
>
> Ben.
>
> --
> Ben Hutchings
> The generation of random numbers is too important to be left to chance.
> - Robert Coveyou

If gcc maintainers agree include a dummy gcc-source-armel package with
Depends: gcc-source (= 1.2-3). That way the cross build package will
require the right source. It ensures they always enter/leave testing
as a pair.

MfG
Goswin

--
To UNSUBSCRIBE, email to debian-devel-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Goswin von Brederlow · External Since: Feb 09, 2009 Posts: 90

Hector Oron <hector.oron RemoveThis @gmail.com> writes:

> Hello,
>
> I would like to do a little explanation on the ITP I have filled for
> {linux,binutils,eglibc,gcc-4.3,gcc-4.4,gdb}-armel.
>
> These set of packages provide a cross toolchain for armel targets to
> be built on i386 and amd64 platforms (maybe ppc could be added)
>
> In order to avoid code duplication in the archive, this packages
> build depend on -source packages.
>
> As major technical issues, I would try to build cross compilers
> with --sysroot support, but that means dpkg-cross need to be updated
> for sysroot paths. For now, we might take the road we have been doing
> at emdebian.org (for many years) and start changing bits towards a
> nice sysrooted solution.
>
> Please, let me know if you need farther explanations on some topics
> or if you have any comment.
>
> Kind regards,

Why do you need --sysroot support? Or what prevents a --sysroot of /
when using the multiarch directories?

It seems like wasted work with multiarch being a release goal for
squeeze. Hop on the wagon and make it work for you too.

MfG
Goswin

--
To UNSUBSCRIBE, email to debian-devel-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Mark Hymers · External Since: Apr 08, 2007 Posts: 13

On Mon, 02, Nov, 2009 at 12:43:42PM +0000, Philipp Kern spoke thus..
> Of course it is a sane approach but very special care needs to be taken when
> releasing to ensure GPL compliance. So what we should get is support in the
> toolchain to declare against what source package the upload was built to
> keep that around.

We haven't implemented that yet for the archive software but it's on the
TODO list (and not that difficult). None of us have had time to do the
d-d-a post from the ftpteam meeting yet, but I'll make sure information
is in there about it.

I'm hoping to the archive-side support done in the next week or so.

Cheers,

Mark

--
Mark Hymers <mhy at debian dot org>

"Oh, this is John Reid who is 'Cabinet Bruiser' which just means that he's
a bit squat, ugly and unpleasant and therefore gets to be called a 'Bruiser'."
Jeremy Hardy, The News Quiz

--
To UNSUBSCRIBE, email to debian-devel-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Florian Weimer · External Since: Nov 10, 2004 Posts: 648

* Ben Hutchings:

> You can disagree all you like, but I believe that the FTP team will
> currently reject any new packages that use source code from their build-
> dependencies.

Surely this is not true because that would rule out many programs
written in C++.

--
To UNSUBSCRIBE, email to debian-devel-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Hector Oron · External Since: Jul 28, 2009 Posts: 6

Hello,

2009/11/2 Goswin von Brederlow <goswin-v-b RemoveThis @web.de>:
> Why do you need --sysroot support? Or what prevents a --sysroot of /
> when using the multiarch directories?
>
> It seems like wasted work with multiarch being a release goal for
> squeeze. Hop on the wagon and make it work for you too.

As you already know, multiarch does not have a plan (yet) for -dev
packages. In terms of cross compiling, things are unclear which way is
painless sysroot or multiarch (orthogonal solutions). The more I
think, the more I believe sysroot is painless.

But anyway, maybe the solution is having both kind of compilers, so is
the user the one that chooses it or maybe we can continue walking
until multiarch shows its head.

--
Héctor Orón

--
To UNSUBSCRIBE, email to debian-devel-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Goswin von Brederlow · External Since: Feb 09, 2009 Posts: 90

Hector Oron <hector.oron.RemoveThis@gmail.com> writes:

> Hello,
>
> 2009/11/2 Goswin von Brederlow <goswin-v-b.RemoveThis@web.de>:
>> Why do you need --sysroot support? Or what prevents a --sysroot of /
>> when using the multiarch directories?
>>
>> It seems like wasted work with multiarch being a release goal for
>> squeeze. Hop on the wagon and make it work for you too.
>
> As you already know, multiarch does not have a plan (yet) for -dev
> packages. In terms of cross compiling, things are unclear which way is
> painless sysroot or multiarch (orthogonal solutions). The more I
> think, the more I believe sysroot is painless.

Your cross-libfoo-dev package would have to solve this. Usualy the
cross-libfoo-dev package would only contain the *.so link in
/usr/lib/arch-os-libc/ and depend on the normal -dev package. Only
when the library has architecture specific include files (not in the
architecture specific dir) would you need to move header files
around. Short term in dpkg-cross, long term in the debian package
itself.

For multiarch the problematic part is actually just splitting out the
*.so links into an arch:any package and common header files into an
arch:all package. A lot of work to modify rules files and lots of new
tiny debs for the archive (and NEW processing). So it got put off for
later. Your dpkg-cross can automatically do this in 99% of cases
leaving verry little handholding.

> But anyway, maybe the solution is having both kind of compilers, so is
> the user the one that chooses it or maybe we can continue walking
> until multiarch shows its head.
>
> --
> Héctor Orón

MfG
Goswin

--
To UNSUBSCRIBE, email to debian-devel-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Hector Oron · External Since: Jul 28, 2009 Posts: 6

Hello,

2009/11/4 Goswin von Brederlow <goswin-v-b DeleteThis @web.de>:
> Neil Williams <codehelp DeleteThis @debian.org> writes:

While being highly interesting talk to me, this discussion is no
relevant to the ITP. I would suggest to either fork the thread or
discuss at debian-embedded DeleteThis @l.d.o

Thanks ! I appreciate your comments.

--
Héctor Orón

--
To UNSUBSCRIBE, email to debian-devel-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Loïc · External Since: Nov 11, 2004 Posts: 475

Hi

(We also had separate discussions Hector, Matthias and I, sometimes
including debian-embedded@ and/or Neil Williams -- I'm catching up on
the debian-devel@ thread.)

On Sun, Nov 01, 2009, Hector Oron wrote:
> I would like to do a little explanation on the ITP I have filled for
> {linux,binutils,eglibc,gcc-4.3,gcc-4.4,gdb}-armel.

Another approach which Matthias suggested (IIRC) is to use a single
source package build-depending on all -source packages to build the
cross-toolchain with a single upload. With the split sources approach,
you need something like 7 uploads to bootstrap the toolchain (I'm not
counting gdb), and a close number if any of linux, binutils, eglibc or
gcc change. A single source package would be very heavy to build, but
would be more effective.

I worked on a linux/binutils/eglibc/gcc as separate source packages
implementation recently, based on the Ubuntu toolchain (basically
identical to Debian's for this discussion). The resulting tree is at:
bzr co lp:~lool/+junk/cross-toolchain
and some binary packages for armel are available in my PPA, albeit
these still suffer from a couple of important issues.

> In order to avoid code duplication in the archive, this packages
> build depend on -source packages.

I don't actually like the amount of logic/duplication with the
toolchain source I ended up with; instead, it would make more sense to
extend the rules in the real packages (gcc, eglibc, binutils etc.) to
be able to produce first stage, second stage etc. cross-compilers and
call them from a new cross-toolchain package. I would also like to
propose to change the -source packages to ship original .dsc, .diff.gz
etc. instead of a patched or unpatched tarball with or without patches
and/or rules to apply the patches.

> As major technical issues, I would try to build cross compilers
> with --sysroot support, but that means dpkg-cross need to be updated
> for sysroot paths. For now, we might take the road we have been doing
> at emdebian.org (for many years) and start changing bits towards a
> nice sysrooted solution.

As we already discussed in private email, I find sysroot an interesting
path forward, especially knowing that --with-headers/--with-libs is
deprecated upstream, but I believe it's an orthogonal change which
should be implemented in the toolchain as time permits. I would also
prefer relocatable toolchains, but using the cross-compiler locations
is fine for now.

I'd love to help on this cross-toolchain project for Debian and
Ubuntu's uses and will look at discussing it with Matthias and other
interested parties at UDS next week.

Thanks,
--
Loïc Minier

--
To UNSUBSCRIBE, email to debian-devel-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Loïc · External Since: Nov 11, 2004 Posts: 475

On Mon, Nov 02, 2009, Goswin von Brederlow wrote:
> If gcc maintainers agree include a dummy gcc-source-armel package with
> Depends: gcc-source (= 1.2-3). That way the cross build package will
> require the right source. It ensures they always enter/leave testing
> as a pair.

I think that would be better done in the cross toolchain package
instead.

However, I do not like the idea of enforcing GPL requirements with
exact version in the cross toolchain when the archive doesn't support
it for any package yet. I think pointing at a snapshot.debian.net
alike service providing all versions of all sources which have ever
been in Debian unless they can't be redistributed is a better way of
solving the source compliance problem and it would actually solve it in
the cross toolchain case as well.

--
Loïc Minier

--
To UNSUBSCRIBE, email to debian-devel-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Goswin von Brederlow · External Since: Feb 09, 2009 Posts: 90

Loïc Minier <lool.TakeThisOut@dooz.org> writes:

> On Mon, Nov 02, 2009, Goswin von Brederlow wrote:
>> If gcc maintainers agree include a dummy gcc-source-armel package with
>> Depends: gcc-source (= 1.2-3). That way the cross build package will
>> require the right source. It ensures they always enter/leave testing
>> as a pair.
>
> I think that would be better done in the cross toolchain package
> instead.

That was the idea. When building the cross toolchain package it
Build-Depends on gcc-source and then builds both a armel-linux-gnu-gcc
and a gcc-source-armel package with the correct Depends entry.

Since the cross toolchain depends on binutils, glibc and gcc a better
name is probably cross-toolchain-source-armel and a Depends on all 3
source debs. But the idea is the same. The Build-Depends is stuffed
into a Depends and the DAK (and other tools) already knows how to take
care of that. And it will always match exactly the specific versions
used to actualy build the binaries.

> However, I do not like the idea of enforcing GPL requirements with
> exact version in the cross toolchain when the archive doesn't support
> it for any package yet. I think pointing at a snapshot.debian.net

Every single GPL package in Debian does have this through its source
package. It is only the special case of the cross toolchain that
breaks it. The dummy package with Depends is a simple way to ensure
GPL compliance without having to change the DAK to track Build-Depends
all of a sudden.

If/when the DAK can track Build-Depends: gcc-source itself, and I
don't see how the DAK should know what gcc-source the buildd actualy
did use, the dummy package could be dropped again. But till then the
extra package doesn't hurt. It is tiny and trivial to generate.

> alike service providing all versions of all sources which have ever
> been in Debian unless they can't be redistributed is a better way of
> solving the source compliance problem and it would actually solve it in
> the cross toolchain case as well.

That does not solve the problem for any distributor, esspecially
CD/DVD images. If you do not ship the source with the binary then you
need to give a written 3 year garanty for the source or pass one
along. Debian does not give that so distributors can not pass it
along. And you can not expect them to mirror snapshot.debian.net.

MfG
Goswin

--
To UNSUBSCRIBE, email to debian-devel-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org