Help!

Please test dpkg with multiarch support

 
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9
Post new topic   General Reply to Topic (not reply to a specific post)    Forums Home -> Development RSS
Next:  Accepted python-visual 1:5.12-1.4 (source amd64)  
Author Message
Jakub Wilk
External


Since: Feb 09, 2010
Posts: 491



PostPosted: Fri Feb 10, 2012 5:10 pm    Post subject: Re: Please test gzip -9n - related to dpkg with multiarch support [Login to view extended thread Info.]
Archived from groups: linux>debian>devel (more info?)

* Guillem Jover , 2012-02-09, 03:45:
>>But anyway, I believe that in the long run we should simply deprecate
>>compressing stuff in /usr/share/doc/.
>
>So the main reason people are arguing for shared files boils down to
>used size, either in installed files, or Packages files, etc,

I don't know what is the "main" reason for other people. (But I doubt
it's about saving space as you're trying to imply.)

>yet you want to fix the compression issue by not compressing them and
>using even more space?

Is that surprising to you that different people can advocate the very
same thing for different reasons?

I think that compressing documentation is an anachronism, which we
should get rid of regardless of whether it helps multi-arch or not.

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120210163034.GA3133@jwilk.net
Back to top
Ian Jackson
External


Since: Jan 03, 2007
Posts: 95



PostPosted: Fri Feb 10, 2012 6:10 pm    Post subject: Re: Please test gzip -9n - related to dpkg with multiarch support [Login to view extended thread Info.]
Archived from groups: linux>debian>devel, others (more info?)

Bastian Blank writes ("Re: Please test gzip -9n - related to dpkg with multiarch support"):
>On Thu, Feb 09, 2012 at 11:45:52AM -0400, Joey Hess wrote:
>>And then if I have a multiarch system, and want to locally download the
>>source of some library, build it and install it, dpkg will complain if I
>>didn't use the same gzip that was used to build other arch versions I
>>have installed.
>
>dpkg would complain anyway, because the versions are different.

(a) You could make them not be. Eg just download the existing source
code and tweak it and rebuild and install.

(b) There will surely be some --force- option you can use to override
this and it should not be necessary to also override problems
involving different versions of the same files.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20277.20318.397042.776523@chiark.greenend.org.uk
Back to top
Russ Allbery
External


Since: Nov 17, 2005
Posts: 1209



PostPosted: Fri Feb 10, 2012 7:10 pm    Post subject: Re: Please test gzip -9n - related to dpkg with multiarch support [Login to view extended thread Info.]
Archived from groups: linux>debian>devel (more info?)

Steve Langasek writes:

> And what about adding 700 packages vs. adding no packages at all, in the
> case of systems which aren't going to have multiarch enabled?

> This would impact systems of all archs, not just those for which multiarch
> is a significant use case.

I'm having a really hard time getting excited about 700 packages. I
haven't checked the numbers, but I would be surprised if we don't add more
than twice that number during normal development between stable releases.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/87y5saef5l.fsf@windlord.stanford.edu
Back to top
Guillem Jover
External


Since: Nov 13, 2004
Posts: 674



PostPosted: Sat Feb 11, 2012 12:10 am    Post subject: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

[ Obviously this "summary" could be considered biased, but I do think
the facts presented are accurate. ]

Hi,

The two reasons for the shared / reference counted files (refcnt from
now on) implementation in dpkg have been:

* To avoid massive package proliferation (due to the mandated copyright
and changelog files), thus the work involved in a one time split and
the size increase in Packages indices.

* To avoid unneeded file duplication, thus wasted space (due to those
mandated files, but also partially just as a consequence of not
splitting files into new arch:all packages, per above).


This has the following implications:

* Deploying refcnt means that M-A:same packages must always be at the
same exact installed version, so that the file contents can match.

More difficult upgrade paths, as this ties the different arch
dependency trees around M-A:same barriers.

* binNMUs need to be performed in lockstep for *all* architectures,
because the installed versions need to match.

Causing useless buildd usage and user downloads for arches not
affected. "Fixing" this by making dpkg treat binNMU versions specially,
besides being just another special case needed for M-A:same packages,
would be wrong, as arch-indep content can actually change between
builds, ex. generated documentation.

* binNMUs for the same version might not be co-installable because doc
generators, compressors, etc, might not always produce the same output.

This is a pretty fragile thing to rely on. New architectures or local
builds might give a hard time if generated output changed in the past.
A possible fix, but only for the compressed files case might be to ship
them uncompresesd, but that counters the desire to reduce wasted space.

* binNMUs for the same version cannot be co-installed anyway as their
changelogs differ.

That could be "fixed" by using the same email address and a hardcoded
date, or not including the binNMU entry at all, or moving that entry
to a new field, etc. All of which seem like ugly hacks, or a possible
loss of information.

* It means special casing M-A:same on indentical file conflict.

The same thing could be argued to be made possible for packages
generated from the same source, a "problem" we've always had and
managed just fine up to now with changes at the packaging level.

* Once implemented, this "feature" cannot be taken out, *ever*.

Because it will produce installation errors, and a long transition
would not help because that would not guarantee external or old
packages are fine.


Conclusion
----------

The above means that binNMUs will be currently unusable for any source
package building an M-A:same package, making the release team's job
harder, or requiring sourceful uploads by maintainers instead.

Given the numbers seen on this thread, the estimated amount of new
required packages to be split off is actually pretty low (less than %2
of the current total), and new arch:all packages should be actually
considered cheap as long as the payload weighs more than the metadata
and the binary format itself, and they should generally actually reduce
archive and multiarch DVD space usage; for Packages indices there's
pdiffs which (although not currently optimally implemented) should only
get downloaded on specific package updates and Descriptions are only
downloaded once nowadays. And these are nothing compared to the amount
of new packages pulled in per each foreign arch configured.

It's been mentioned that splitting packages is a daft idea because
it causes more burden to library package maintainers while dpkg could
do the job once instead, but this is a progressive one time thing, while
the above implications are *forever*, and if maintainers are required to
do sourceful uploads instead of getting binNMUs done it actually means
it's going to be even more of a burden for them.

Even if no packages were to get split off and all arch-indep file
paths be arch-qualified (which would actually be wrong in some cases as
some of those arch-indep files should not get an arch-qualified path),
and the overhead of the duplicated files was considered an issue,
(although the actual libraries will usually use way more space than
those few duped files), there's always --path-exclude for the ones not
affecting functionality.

It does not seem to make sense to consider the "huge" space usage due
to not refcnt'ing an issue, when for that to happen and be significant
one would need to install hundreds of M-A:same packages for multiple
architectures, taking hundreds of MiB (if not GiB), at which point I'm
not sure how one can make a fuss over some hundred wasted MiB, if at
all.

For the unreliable generated output problem, even if gzip is to be
considered frozen and in maintenance mode now, that does not mean this
could not change in the future. It also means we cannot safely consider
switching compressors in the future, as we have cornered ourselves by
the design.

Switching to uncompressed files to workaround the unreliable generated
output problem, still only papers over one part of the issue, and defeats
the size savings in common situations (single arch installs).

----

So it really does not seem worth it, it does way more harm than good,
it will generate more overall waste, make transitions and upgrades more
difficult, it makes the M-A:same packages even more asymmetric and
exceptional than they need to be and the size reduction arguments do
not really seem to hold too much, and seems to be the actual overall
more complex solution to the problem.

In addition concerning the mandatory files (copyright and changelog), if
we'd eventually go forward with my proposal to make them actual package
metadata, then dpkg can actually manage them in its db in any way we see
fit, including automatically compressing or refcnt'ing them for example
when they actually match, and as such reducing installed size usage.

Given all the above, I'll be pulling off for now the file refcnt and
version match logic from my pu/multiarch/master branch. If some compelling
arguments are brought up, something I honestly don't really see happening,
then they can be actually reintroduced at any point.


Proposed solution
-----------------

M-A:same packages cannot have any conflicting files with their foreign
counterparts. Thus:

For files in M-A:same packages under a pkgname based path, the pkgname
should always be arch-qualified with the Debian architecture. Most of
these could be handled automatically by debhelper and cdbs, this includes
things like:

/usr/share/doc/pkgname/
/usr/share/bug/pkgname
/usr/share/lintian/overrides/pkgname
/usr/share/mime-info/pkgname.*
/usr/share/menu/pkgname
...

(Joey, I'm guessing you might consider it too late to do some of these in
debhelper for compat level 9, right?)

For toolchain related files on M-A:same packages, their path should get
arch-qualified using the multiarch triplet, this includes arch-dependent
headers and similar.

The remaining files that are truly arch-independent, like headers, man
pages, development docs, etc, should be split into arch:all package(s),
to something along these lines:

libfooN-doc
libfooN-headers
libfooN-common
libfooN-common-dev
libfooN-data
girX.Y-foo
...

Anything else remaining should be considered a bug.


regards,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120210225620.GA8782@gaara.hadrons.org
Back to top
Jakub Wilk
External


Since: Feb 09, 2010
Posts: 491



PostPosted: Sat Feb 11, 2012 1:10 am    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

* Guillem Jover , 2012-02-10, 23:56:
>[ Obviously this "summary" could be considered biased, but I do think
> the facts presented are accurate. ]

Well, biased in an euphemism here...

>The two reasons for the shared / reference counted files (refcnt from
>now on) implementation in dpkg have been:
>
>* To avoid massive package proliferation (due to the mandated copyright
>and changelog files), thus the work involved in a one time split and
>the size increase in Packages indices.
>
>* To avoid unneeded file duplication, thus wasted space (due to those
>mandated files, but also partially just as a consequence of not
>splitting files into new arch:all packages, per above).

Personally, I don't consider any of these reasons very important.

How about:
* Because this the obvious and elegant way of doing things. It makes
multiarchification easy for packagers, and invisible for uses, including
those users who don't care about multi-arch (unless they rely on paths
to the libraries, which they never should do).

>* Deploying refcnt means that M-A:same packages must always be at the
>same exact installed version, so that the file contents can match.
>↓
>More difficult upgrade paths, as this ties the different arch
>dependency trees around M-A:same barriers.

By allowing co-installation of two different versions of the same
package, you are opening a can of worms, regardless of whether refcnt is
implemented or not.

>* binNMUs need to be performed in lockstep for *all* architectures,
>because the installed versions need to match.
>↓
>Causing useless buildd usage and user downloads for arches not
>affected. "Fixing" this by making dpkg treat binNMU versions specially,
>besides being just another special case needed for M-A:same packages,

What do you mean by "another"? Yes, MA:same packages needs special
treatment, because they _are_ special.

>* binNMUs for the same version cannot be co-installed anyway as their
>changelogs differ.
>↓
>That could be "fixed" by using the same email address and a hardcoded
>date, or not including the binNMU entry at all, or moving that entry to
>a new field, etc. All of which seem like ugly hacks, or a possible loss
>of information.

http://lists.debian.org/debian-devel/2012/02/msg00316.html

>* Once implemented, this "feature" cannot be taken out, *ever*.

This boils down to "I don't like it". If it's useful, why would one
consider taking it out?

>Proposed solution
>-----------------
>
>M-A:same packages cannot have any conflicting files with their foreign
>counterparts. Thus:
>
>For files in M-A:same packages under a pkgname based path, the pkgname
>should always be arch-qualified with the Debian architecture. Most of
>these could be handled automatically by debhelper and cdbs, this
>includes things like:
>
> /usr/share/doc/pkgname/
> /usr/share/bug/pkgname
> /usr/share/lintian/overrides/pkgname
> /usr/share/mime-info/pkgname.*
> /usr/share/menu/pkgname
> ...

This will require changes to the Policy, to which I (and hopefully other
developers) will object.

Please don't throw into the mud work of individual developers (including
me) who already converted their packages to multi-arch. Thank you very
much.

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211001446.GB2797@jwilk.net
Back to top
Jonathan Nieder
External


Since: Feb 26, 2009
Posts: 694



PostPosted: Sat Feb 11, 2012 2:10 am    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

Hi,

Jakub Wilk wrote:

> How about:
> * Because this the obvious and elegant way of doing things. It makes
> multiarchification easy for packagers, and invisible for uses,
> including those users who don't care about multi-arch (unless they
> rely on paths to the libraries, which they never should do).

I suppose that is a matter of opinion. The need to make sure files
that are not arch-qualified match at least functionally between
architectures is subtle and not very easy. Tying package versions in
different architectures, which makes dependency chains more rigid, is
not invisible to users.

However, in the special case of header files (which are text files
that almost never vary between architectures), I agree that it would
be nice to be able to preserve the simplicity of the current approach
from the packager's perspective.

[...]
> By allowing co-installation of two different versions of the same
> package, you are opening a can of worms, regardless of whether
> refcnt is implemented or not.

Could you elaborate on this?

As long as dependencies are accurate, I don't see how allowing
co-installation of the same package for two different architectures at
different versions is any more complicated than pinned to the same
version.

[...]
>> * Once implemented, this “feature” cannot be taken out, *ever*.
>
> This boils down to “I don't like it”. If it's useful, why would one
> consider taking it out?

After trying out the approach without refcounted files for a while, it
would be painless for the project to say "This is too much trouble"
and add refcounting. By contrast, it is very painful to move in the
other direction. I think that is worth taking into account.

[...]
>> Proposed solution
[...]
> This will require changes to the Policy, to which I (and hopefully
> other developers) will object.

Last time I checked, multiarch is not in policy yet.

> Please don't throw into the mud work of individual developers
> (including me) who already converted their packages to multi-arch.

I agree that the extra work of removing "multi-arch: same" for
existing -dev packages that have been converted is a major downside.
And on the other hand, the need throughout Debian infrastructure to
support the very fragile refcount approach would be a downside to that
approach.

This is not so one-sided as you seem to be suggesting.

Jonathan
who wishes he knew of a fifth approach without the downsides of the
others proposed so far Smile


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211005559.GA32671@burratino
Back to top
Riku Voipio
External


Since: Nov 10, 2004
Posts: 167



PostPosted: Sat Feb 11, 2012 2:10 am    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On Fri, Feb 10, 2012 at 11:56:20PM +0100, Guillem Jover wrote:
> [ Obviously this "summary" could be considered biased, but I do think
> the facts presented are accurate. ]

> The two reasons for the shared / reference counted files (refcnt from
> now on) implementation in dpkg have been:

Well the bias comes up quite quick when you conveniently omit the MAIN
reason right here and rather discuss it at the end of document..

* Because shared files packaging simpler for maintainers.

Package profiliation and duplication of arch-independent data are merely
effecty that happen when packagers workaround the lack of shared identical
files.

> * To avoid massive package proliferation (due to the mandated copyright
> and changelog files), thus the work involved in a one time split and
> the size increase in Packages indices.
>
> * To avoid unneeded file duplication, thus wasted space (due to those
> mandated files, but also partially just as a consequence of not
> splitting files into new arch:all packages, per above).

> This has the following implications:
>
> * Deploying refcnt means that M-A:same packages must always be at the
> same exact installed version, so that the file contents can match.
> ↓
> More difficult upgrade paths, as this ties the different arch
> dependency trees around M-A:same barriers.
>
> * binNMUs need to be performed in lockstep for *all* architectures,
> because the installed versions need to match.
> ↓
> Causing useless buildd usage and user downloads for arches not
> affected. "Fixing" this by making dpkg treat binNMU versions specially,
> besides being just another special case needed for M-A:same packages,
> would be wrong, as arch-indep content can actually change between
> builds, ex. generated documentation.
>
> * binNMUs for the same version might not be co-installable because doc
> generators, compressors, etc, might not always produce the same output.
> ↓
> This is a pretty fragile thing to rely on. New architectures or local
> builds might give a hard time if generated output changed in the past.
> A possible fix, but only for the compressed files case might be to ship
> them uncompresesd, but that counters the desire to reduce wasted space.
>
> * binNMUs for the same version cannot be co-installed anyway as their
> changelogs differ.
> ↓
> That could be "fixed" by using the same email address and a hardcoded
> date, or not including the binNMU entry at all, or moving that entry
> to a new field, etc. All of which seem like ugly hacks, or a possible
> loss of information.

One solution for the binNMU changelogs and generated docs would be to
use arch-qualified paths for those files. That is much more lightweight
solution that arch-qualifying all files, even if identical.

> Given all the above, I'll be pulling off for now the file refcnt and
> version match logic from my pu/multiarch/master branch. If some compelling
> arguments are brought up, something I honestly don't really see happening,
> then they can be actually reintroduced at any point.

> Proposed solution
> -----------------

> M-A:same packages cannot have any conflicting files with their foreign
> counterparts. Thus:

> For files in M-A:same packages under a pkgname based path, the pkgname
> should always be arch-qualified with the Debian architecture. Most of
> these could be handled automatically by debhelper and cdbs, this includes
> things like:
>
> /usr/share/doc/pkgname/
> /usr/share/bug/pkgname
> /usr/share/lintian/overrides/pkgname
> /usr/share/mime-info/pkgname.*
> /usr/share/menu/pkgname
> ...

I find the requring arch-qualified path for for arch-independent
data ugly system architecture. But of course the beauty of architecture is in
the eye of the beholder (and lets not forget that Unix with its worse-is-better
philosophy[1] was never intended to be architectural masterpiece).

Personally if leaving out shared files makes you upload multiarch enabled
dpkg to unstable before sagrada familia is complete, i can live it (cursing
silently in my room converting packages to the new requirement...). I can
take the trade-off of having something better-than-current soon over having
the perfect version in distant future...

Riku


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211012833.GA21185@afflict.kos.to
Back to top
Russ Allbery
External


Since: Nov 17, 2005
Posts: 1209



PostPosted: Sat Feb 11, 2012 2:10 am    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

Steve Langasek writes:
> On Fri, Feb 10, 2012 at 06:56:00PM -0600, Jonathan Nieder wrote:

>> I agree that the extra work of removing "multi-arch: same" for existing
>> -dev packages that have been converted is a major downside. And on the
>> other hand, the need throughout Debian infrastructure to support the
>> very fragile refcount approach would be a downside to that approach.

> Which infrastructure do you have in mind? I don't see anything that's
> needed besides a dpkg implementation (which we have) and some tools to
> sanity-check coinstallability (which 1. we would need anyway, and 2. we
> also have a preliminary implementation of).

We need a guarantee that gzip will always produce the same output, which
we already know isn't the case and which doesn't look sustainable going
forward. That's looking rather uncomfortable. The preliminary results
there point to that being somewhat problematic.

I think we have to do something saner with changelog files eventually
regardless, but I'm curious: how did Ubuntu deal with the binNMU problem
that Guillem identified? If you binNMU a library on amd64 but not on
i386, as near as I can tell that's going to make the library not work for
multiarch until the next sourceful upload, no? I think Ubuntu has
binNMUs; haven't you run into this issue?

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/87zkcqrw2w.fsf@windlord.stanford.edu
Back to top
Adam D. Barratt
External


Since: Feb 14, 2009
Posts: 336



PostPosted: Sat Feb 11, 2012 11:10 am    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On Fri, 2012-02-10 at 17:35 -0800, Russ Allbery wrote:
> I think we have to do something saner with changelog files eventually
> regardless, but I'm curious: how did Ubuntu deal with the binNMU problem
> that Guillem identified? If you binNMU a library on amd64 but not on
> i386, as near as I can tell that's going to make the library not work for
> multiarch until the next sourceful upload, no? I think Ubuntu has
> binNMUs; haven't you run into this issue?

Unless something's changed recently, Ubuntu doesn't have binNMUs in the
same way Debian does. Instead, they have "no change" source uploads
with rebuilds on all architectures. There have been discussions of
introducing binNMUs - see
https://bugs.launchpad.net/launchpad/+bug/245594 for example. It wasn't
entirely clear to me from the surrounding discussions I found whether
the remaining reasons that it hasn't been implemented are technical or
simply because no-one's done the work.

Regards,

Adam


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/1328953563.27786.8.camel@jacala.jungle.funky-badger.org
Back to top
Raphael Hertzog
External


Since: May 28, 2005
Posts: 714



PostPosted: Sat Feb 11, 2012 11:10 am    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

Hello,

On Sat, 11 Feb 2012, Jakub Wilk wrote:
> >* Deploying refcnt means that M-A:same packages must always be at
> >the same exact installed version, so that the file contents can
> >match.
> >↓
> >More difficult upgrade paths, as this ties the different arch
> >dependency trees around M-A:same barriers.
>
> By allowing co-installation of two different versions of the same
> package, you are opening a can of worms, regardless of whether
> refcnt is implemented or not.

I'm tempted to agree but I can't come up with a good reason.

The most worrying problem would be a version skew between
2 instances of libfoo and its common libfoo-data.

It could be a source of subtle bugs if this leads to having
libfoo:i386 (1.0) + libfoo:amd64 (2.0) + libfoo-data:all (2.0)

But then the proper answer is for the maintainer to put
a tight dependency "Depends: libfoo-data (= ${source:Version})".

Any other problem is nothing else than a usual dependency problem
that would likely also be triggered in a non-multiarch world. Or am
I missing something? Do you have examples of possible problems?

> >* binNMUs need to be performed in lockstep for *all*
> >architectures, because the installed versions need to match.
> >↓
> >Causing useless buildd usage and user downloads for arches not
> >affected. "Fixing" this by making dpkg treat binNMU versions
> >specially, besides being just another special case needed for
> >M-A:same packages,
>
> What do you mean by "another"? Yes, MA:same packages needs special
> treatment, because they _are_ special.

To some extent... in any case I agree that we should at some point do
something to allow bin-nmu of packages on some arches only and also to
allow co-installation of packages with versions differing only by their
binnmu number (this could be easily fixed by using version of the source
package intead of binary version).

I'm not convinced that Guillem's answer is the right one though. Your
suggestion below seams appealing too.

> http://lists.debian.org/debian-devel/2012/02/msg00316.html

Quoting you:

Packages could simply split debian/changelog into two parts:
/u/s/d/$p/changelog(.Debian).gz - the architecture-independent part;
/u/s/d/$p/changelog.binNMU-$arch.gz - the binNMU part.

Cheers,
--
Raphal Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211101538.GM13299@rivendell.home.ouaza.com
Back to top
Jakub Wilk
External


Since: Feb 09, 2010
Posts: 491



PostPosted: Sat Feb 11, 2012 12:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

* Joey Hess , 2012-02-10, 22:35:
>> /usr/share/doc/pkgname/
>> /usr/share/bug/pkgname
>> /usr/share/lintian/overrides/pkgname
>> /usr/share/mime-info/pkgname.*
>> /usr/share/menu/pkgname
>> ...
>>
>> (Joey, I'm guessing you might consider it too late to do some of these in
>> debhelper for compat level 9, right?)
>
>I wouldn't worry about this, there's still time to make whatever
>changes might be needed in v10. It's not actually clear this would be a
>change that needs a compat level at all.

Why is that not clear?

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211104357.GA2837@jwilk.net
Back to top
Henrique de Moraes Holsch
External


Since: Nov 11, 2004
Posts: 732



PostPosted: Sat Feb 11, 2012 1:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On Sat, 11 Feb 2012, Raphael Hertzog wrote:
> It could be a source of subtle bugs if this leads to having
> libfoo:i386 (1.0) + libfoo:amd64 (2.0) + libfoo-data:all (2.0)
>
> But then the proper answer is for the maintainer to put
> a tight dependency "Depends: libfoo-data (= ${source:Version})".

Err, that's heavily frowned because it breaks binNMUs if libfoo-data is
arch:all, so we must decide on what the best way to deal with it is, and
update best practice and policy accordingly.

Too bad we don't have "lightweight sub-packages". That would just kill the
need for multiarch-same and avoid a lot of nasty issues. You'd shunt all
arch-dependent files to the arch-dep subpackage. Then we'd just have to
decide whether we'd allow binNMUs of subpackages, or do it the Ubuntu way
(which basically boils down to how painful it would be for the
smaller/slower autobuilders to switch from binNMUs to no-source-changes
rebuilds on all arches).

--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211114746.GA772@khazad-dum.debian.net
Back to top
Jakub Wilk
External


Since: Feb 09, 2010
Posts: 491



PostPosted: Sat Feb 11, 2012 1:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

* Raphael Hertzog , 2012-02-11, 11:15:
>>>* Deploying refcnt means that M-A:same packages must always be at the
>>>same exact installed version, so that the file contents can match.
>>>↓
>>>More difficult upgrade paths, as this ties the different arch
>>>dependency trees around M-A:same barriers.
>>
>>By allowing co-installation of two different versions of the same
>>package, you are opening a can of worms, regardless of whether refcnt
>>is implemented or not.
>
>I'm tempted to agree but I can't come up with a good reason.

I did have some scary scenarios in mind when I wrote this, but it was
apparently because:
a) it was middle of the night;
b) "let's arch-qualify everything" didn't fit well my mental model.

So please disregard what I wrote in this point. Sorry for the noise.


It shall be noted that if we allow co-installation of different
versions, then we should disallow sharing files not only shipped
directly in .deb, but also created by maintainer scripts.

Some MA:same packages currently try to share data created by maintainer
scripts, and probably (almost?) all of them do it wrong. See #647428 for
an example.

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211115057.GA8401@jwilk.net
Back to top
Henrique de Moraes Holsch
External


Since: Nov 11, 2004
Posts: 732



PostPosted: Sat Feb 11, 2012 1:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On Sat, 11 Feb 2012, Henrique de Moraes Holschuh wrote:
> On Sat, 11 Feb 2012, Raphael Hertzog wrote:
> > It could be a source of subtle bugs if this leads to having
> > libfoo:i386 (1.0) + libfoo:amd64 (2.0) + libfoo-data:all (2.0)
> >
> > But then the proper answer is for the maintainer to put
> > a tight dependency "Depends: libfoo-data (= ${source:Version})".
>
> Err, that's heavily frowned because it breaks binNMUs if libfoo-data is
> arch:all, so we must decide on what the best way to deal with it is, and
> update best practice and policy accordingly.

Yikes, never mind. ${source:version} exists exactly to avoid the above
problem. Sorry about this.

--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211115325.GB772@khazad-dum.debian.net
Back to top
Jakub Wilk
External


Since: Feb 09, 2010
Posts: 491



PostPosted: Sat Feb 11, 2012 1:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

* Jonathan Nieder , 2012-02-10, 18:56:
>This is not so one-sided as you seem to be suggesting.

Yes. I apologize for implying this.

>Jonathan
>who wishes he knew of a fifth approach without the downsides of the
>others proposed so far Smile

Let me try:

1. We allow sharing files between architectures.

2. If
- package $pkg (of architecture $arch1) is already installed and,
- package $pkg (of architecture $arch2) is going to be installed and,
- $pkg:$arch1 and $pkg:$arch2 both ship a $file with _different_
hash then dpkg would rename $arch2's $file into $file.dpkg-$arch2, using
a mechanism similar to diversions.

If $pkg:$arch1 is later removed, dpkg would rename files with
..dpkg-$archN suffix back to original names

3. We will teach debsums about these "multiarch diversions".

4. We continue to treat hash mismatches as important (or maybe
serious) bugs, even though they don't prevent package installation
anymore. As a corollary, we still need a separate solution for the
binNMU problem.


(I stole the idea from Ron Lee, and all credit should go to him. But I
didn't consult the details with Ron, so I this proposal doesn't make
sense to you, I shall take all the blame.)

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211123127.GA576@jwilk.net
Back to top
Guillem Jover
External


Since: Nov 13, 2004
Posts: 674



PostPosted: Sat Feb 11, 2012 2:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On Fri, 2012-02-10 at 17:16:29 -0800, Steve Langasek wrote:
> On Fri, Feb 10, 2012 at 06:56:00PM -0600, Jonathan Nieder wrote:
> > As long as dependencies are accurate, I don't see how allowing
> > co-installation of the same package for two different architectures at
> > different versions is any more complicated than pinned to the same
> > version.
>
> I think we're likely to see a lot of bugs introduced in the process of
> making the dependencies accurate if we go this route. So instead of
> refcounting the files, we move them to a new arch: all package. Ok; now,
> suppose some of these files aren't actually architecture-independent:
> they're data but they're generated differently on different architectures,
> and the library expects the data in its native architecture-dependent
> format. (See the parallel thread about "endianness of data files" for
> examples.) Doesn't this proposal eliminate one of our best defenses against
> this packaging error? Having them kicked back as mismatched files, either
> by dpkg or by the archive, seems better to me than letting them land on the
> user's system and break at runtime.

While I agree this is a potential issue, it's not a new one at all or
specific to multiarchified packages, it's actually inherent to every
arch:all package generated from source packages producing arch:any
binaries too, regardless of multiarch.

Instead of treating M-A:same specially on this, I'd rather see a way
to detect this for all current and existing cases instead, if deemed
really necessary.

regards,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211131828.GA19157@gaara.hadrons.org
Back to top
Guillem Jover
External


Since: Nov 13, 2004
Posts: 674



PostPosted: Sat Feb 11, 2012 2:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On Sat, 2012-02-11 at 11:41:58 +0100, Stefano Zacchiroli wrote:
> That is a bug and ought to be fixed in its own right. Then, the
> discussion of how much we should rely on it or not is a different one,
> but it's good to separate the concerns.

As mentioned on the summary, this is not specific to gzip (even if gzip
could pickup development in the future), it is a general problem that
extends to any other compressors and generated files too, for example
docs (POD, html, etc), etc.

guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211132523.GB19157@gaara.hadrons.org
Back to top
Guillem Jover
External


Since: Nov 13, 2004
Posts: 674



PostPosted: Sat Feb 11, 2012 2:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

On Sat, 2012-02-11 at 00:46:53 +0100, Marco d'Itri wrote:
> It is not true that splitting the package is a one time action, every
> release which adds new files will require dealing with the split.

Assuming a somewhat standard packaging, using debhelper and debian/tmp
or debian/<main-pkg>, where all upstream files get intstalled to, and
selected ones copied/moved over to their debian/<pkg> final destination.
How is this any different from packaging any new upstream release,
regardless of a split due to multiarch?

regards,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120211133449.GC19157@gaara.hadrons.org
Back to top
Russ Allbery
External


Since: Nov 17, 2005
Posts: 1209



PostPosted: Sat Feb 11, 2012 3:10 pm    Post subject: Re: Summary: dpkg shared / reference counted files and version match [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

Guillem Jover writes:
> On Sat, 2012-02-11 at 11:41:58 +0100, Stefano Zacchiroli wrote:

>> That is a bug and ought to be fixed in its own right. Then, the
>> discussion of how much we should rely on it or not is a different one,
>> but it's good to separate the concerns.

> As mentioned on the summary, this is not specific to gzip (even if gzip
> could pickup development in the future), it is a general problem that
> extends to any other compressors and generated files too, for example
> docs (POD, html, etc), etc.

Yeah, generated documentation potentially also varies in annoying ways.
Man pages generated from POD, for example, will be different if the
version of Pod::Man or Pod::Simple used to generate them is different,
which could easily happen across a point release of Perl, and hence across
a binNMU. I suspect Doxygen is even more fragile (although somewhat less
likely to be included directly in a -dev package rather than a separate
-docs package because it's larger).

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/8762fde9hb.fsf@windlord.stanford.edu
Back to top
Goswin von Brederlow
External


Since: Feb 09, 2009
Posts: 303



PostPosted: Sat Feb 11, 2012 5:10 pm    Post subject: Re: Please test gzip -9n - related to dpkg with multiarch support [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)

Steve Langasek writes:

> On Thu, Feb 09, 2012 at 01:40:41PM +0100, Goswin von Brederlow wrote:
>> Steve Langasek writes:
>
>> > - For many of these files, it would be actively harmful to use
>> > architecture-qualified filenames. Manpages included in -dev packages
>> > should not change names based on the architecture; having
>> > /usr/share/pam-config contain multiple files for the same profile, one
>> > for each architecture of the package that's installed, would not work
>> > correctly; etc.
>
>> Appropos pam config. Shouldn't that be arch-qualified (which includes
>> /etc)?
>
> No, it should not. These files are input for a central, shared, common PAM
> configuration meant to be usable by all services on the system. If you have
> a foreign-arch PAM-using service installed, but you don't have the
> foreign-arch versions of the PAM modules that are referenced by
> /etc/pam.d/common-*, that's a bug: the module packages should be installed
> for all archs, not just a subset[1]. The system-level authentication
> configuration should not vary based on the architecture of the binary! And
> if you happen to have a foreign-arch service for which you don't want to use
> the same set of modules, well, your service's config file doesn't have to
> include /etc/pam.d/common-* - but then that's a special case of a service
> that you don't want to use the common config, it's not something we should
> assume by default in multiarch.

Ok, that is acceptable. We just lack any technical means to ensure this
so far. Same problem as for input method plugins for example.

>> Say I have pam modules for ldap installed for amd64 but not for armel?
>
> Why would you do that, except by accident?

MfG
Goswin


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/87aa4pe546.fsf@frosties.localnet
Back to top
Display posts from previous:   
Post new topic   General Reply to Topic (not reply to a specific post)    Forums Home -> Development All times are: Eastern Time (US & Canada)
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9
Page 4 of 9

 
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum