GNU/Linux Java Policy and Packaging

Tom Marble · External Since: May 27, 2006 Posts: 29

All:

I'm very much looking forward to working Java Policy and
Packaging issues at Debconf7.

There has been some great discussion about this recently
from Paul, Manfred, Matthew and others. And, of course,
there is a lot of documentation out there [1].

My proposal is that, if folks agree, that we now actively
list the issues that need to be addressed and work
towards revised (or at least new documentation for) Java Policy.
These start at the very practical level and move to more
generic -- pan-Linux Java Policy. We had some great
discussions along these lines at the most recent
DevJam at FOSDEM 2007 [2]. The overriding goal here is
that the experience for Java developers, packagers and
users on GNU/Linux is as comfortable as possible
("It Just Works!").

I think the right way to flesh out these ideas is to
work them up on the Wiki, but to get started let
me write some here. I realize most of this is already
decided and works just fine... Again the purpose
is discussion:

Execution
---------
1. Handling alternative runtimes
Gracefully handle the co-existence of many different
runtimes (even multiple versions of one runtime).

2. Insure that the man pages correspond to the
executables (e.g. 'man java' does the right thing
for the current /usr/bin/java alternative).

3. Gracefully handle all the features of a runtime
(e.g. the union of all possible programs in
a runtime, SDK, and also Java Plug-In).
Certain implementations may not have all the
executables (or plugin) and may need to rely
on some default (or error handling) behavior.

4. Support for binfmt_misc?

Administration
--------------
5. Use the priority system for safe "default" behavior.

6. Command line tools for making choices
(e.g. update-java-alternatives).
It would be really nice for there to be global
defaults (priorities), system defaults, and then
user settings (where feasible).

7. GUI tools for making choices
(e.g. a GUI front end for update-java-alternatives).

8. It will be important to have some interface
for runtime argument setting -- esp. for performance.
The challenge here is that tunings (e.g. heap, collector,
compiler, etc) are VM dependent. And the packager's
default may not be a good system wide default.
And users should be able to override these settings
conveniently as well (e.g. for profiling/analysis,
for production environments)

Packaging
---------
9. Debhelper tools to help packaging Java libraries and
applications easier and less error prone (e.g. dh_installjars,
cdbs extensions).

10. Filesystem conventions (FHS) for runtimes (e.g. /usr/lib/jvm)
and libraries (e.g. /usr/share/java).

11. Define virtual provides (free/non-free, which version
of the Java SE platform: 4, 5, 6, 7).

12. Future integration of the the packaging with
the Java Module System (JSR 277).

Upstream and distro Integration
-------------------------------
13. Common upstream watcher. As many distros are interested
in new versions from the same upstream runtimes (and
libraries and apps) it seems that there is an opportunity
for us to collaborate at a community level on
some sort of notification of upstream publication
(i.e. something as simple as Atom/RSS for new versions)

14. Common package decomposition and interdependencies.
Again as many of these applications are identical across
distros (as are the libraries) we may be able to
reduce our community "energy budget" on packaging if
we can share the dependency graph despite packaging
format differences.

15. Common upstream/downstream bug integration.
Ideally if a bug is found in one distro it gets a
tracking bug upstream... and then the upstream bug
can point to all the distro specific bugs.
Perhaps stated more generally -- wouldn't it be
great if searching on "xcb protocol" would
list Java issues on *all* distros?

Petteri Räty has pointed out some of the very interesting
approaches that Gentoo uses in it's java-config-2
structure [2] (I have more documentation that can go on the Wiki).
I'm not proposing that Debian adopt java-config-2 wholesale, but
I think the Gentoo is an excellent example for discussing
different approaches to addressing these challenges.

Regards,

--Tom

[1] some Debian Java documentation

avdyk in 2005
http://wiki.debian.org/CommonJavaPackaging
AldousPenaranda in 2005
http://wiki.debian.org/DebianJavaPackaging
Barry Hawkins in 2006
http://wiki.debian.org/Java/RoadMap
Ola Lundqvist
Stephane Bortzmeyer
http://www.debian.org/doc/packaging-manuals/java-policy/
Debian Java Packaging Project 2006
http://pkg-java.alioth.debian.org/
Debian GNU/Linux Java FAQ. 2006
http://www.debian.org/doc/manuals/debian-java-faq/

[2] http://blogs.sun.com/tmarble/entry/devjam_fosdem_slides

[3] Part of the Gentoo java-config-2 approach
http://rafb.net/p/QxST1h34.html
http://overlays.gentoo.org/proj/java/browser/projects/java-config-2/tr.../src/la

--
To UNSUBSCRIBE, email to debian-java-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Marcus Better · External Since: Nov 19, 2004 Posts: 366

Tom Marble wrote:
> Packaging
> ---------

I think we need to add:

* Library ABI handling and support for parallel installation of multiple
versions (as discussed in another thread [1]).

Marcus

[1] http://thread.gmane.org/gmane.linux.debian.devel.java/6537

--
To UNSUBSCRIBE, email to debian-java-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Petteri Räty · External Since: Jun 14, 2007 Posts: 1

Tom Marble kirjoitti:
>
> I think the right way to flesh out these ideas is to
> work them up on the Wiki, but to get started let
> me write some here. I realize most of this is already
> decided and works just fine... Again the purpose
> is discussion:

I can explain in more detail our system if you want to take the best bits.

>
> Execution
> ---------
> 1. Handling alternative runtimes
> Gracefully handle the co-existence of many different
> runtimes (even multiple versions of one runtime).

checked

betelgeuse@pena ~ $ eselect java-vm list
Available Java Virtual Machines:
[1] blackdown-jdk-1.4.2
[2] ibm-jdk-bin-1.4
[3] ibm-jdk-bin-1.5
[4] kaffe
[5] openjdk-1.7 user-vm
[6] sun-jdk-1.4 system-vm
[7] sun-jdk-1.5
[8] sun-jdk-1.6
[9] sun-jdk-1.6.0.02
[10] sun-jdk-1.7
[11] sun-jre-bin-1.5
[12] sun-jre-bin-1.6

>
> 2. Insure that the man pages correspond to the
> executables (e.g. 'man java' does the right thing
> for the current /usr/bin/java alternative).

betelgeuse@pena ~ $ env | grep MANPATH
MANPATH=/home/betelgeuse/.gentoo/java-config-2/current-user-vm/man:<snip
rest>

>
> 3. Gracefully handle all the features of a runtime
> (e.g. the union of all possible programs in
> a runtime, SDK, and also Java Plug-In).
> Certain implementations may not have all the
> executables (or plugin) and may need to rely
> on some default (or error handling) behavior.

checked:

betelgeuse@pena ~ $ GENTOO_VM="sun-jre-bin-1.6" javac -version
* javac is not available for sun-jre-bin-1.6 on i686
* IMPORTANT: some Java tools are not available on some VMs on some
architectures

>
> 4. Support for binfmt_misc?
>

Was toying with this at some point but should be quite easy to implement
using an init script and some helpers.

>
> Administration
> --------------
> 5. Use the priority system for safe "default" behavior.
>

Seems Debian specific.

>
> 6. Command line tools for making choices
> (e.g. update-java-alternatives).
> It would be really nice for there to be global
> defaults (priorities), system defaults, and then
> user settings (where feasible).
>

checked

>
> 7. GUI tools for making choices
> (e.g. a GUI front end for update-java-alternatives).
>

Would be nice yes.

>
> 8. It will be important to have some interface
> for runtime argument setting -- esp. for performance.
> The challenge here is that tunings (e.g. heap, collector,
> compiler, etc) are VM dependent. And the packager's
> default may not be a good system wide default.
> And users should be able to override these settings
> conveniently as well (e.g. for profiling/analysis,
> for production environments)
>

checked but could be documented better

>
> 10. Filesystem conventions (FHS) for runtimes (e.g. /usr/lib/jvm)
> and libraries (e.g. /usr/share/java).

Shouldn't really matter if you you can use something like java-config to
query the locations but of course Debian needs to have their own
internal policy.

betelgeuse@pena ~ $ java-config -p xalan
/usr/share/xalan/lib/xalan.jar:/usr/share/xalan/lib/serializer.jar

>
> Upstream and distro Integration
> -------------------------------
> 13. Common upstream watcher. As many distros are interested
> in new versions from the same upstream runtimes (and
> libraries and apps) it seems that there is an opportunity
> for us to collaborate at a community level on
> some sort of notification of upstream publication
> (i.e. something as simple as Atom/RSS for new versions)

freshmeat somewhat provides for this
http://meatoo.gentooexperimental.org/

>
> 14. Common package decomposition and interdependencies.
> Again as many of these applications are identical across
> distros (as are the libraries) we may be able to
> reduce our community "energy budget" on packaging if
> we can share the dependency graph despite packaging
> format differences.
>

Well sure dep graph are easy to share but that's not where most of the
time packaging things is spent.

>
> 15. Common upstream/downstream bug integration.
> Ideally if a bug is found in one distro it gets a
> tracking bug upstream... and then the upstream bug
> can point to all the distro specific bugs.
> Perhaps stated more generally -- wouldn't it be
> great if searching on "xcb protocol" would
> list Java issues on *all* distros?
>

I haven't tried it but bugzilla 3.0 should have some features towards
this. But this is not Java specific and KDE/GNOME etc would benefit if
one day something like this is working.

>
> Petteri Räty has pointed out some of the very interesting
> approaches that Gentoo uses in it's java-config-2
> structure [2] (I have more documentation that can go on the Wiki).
> I'm not proposing that Debian adopt java-config-2 wholesale, but
> I think the Gentoo is an excellent example for discussing
> different approaches to addressing these challenges.
>

There is lot's more to our stuff that we didn't cover so feel free to
ask me if you need more info.

Regards,
Petteri
--
Gentoo/Java project lead

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

manfred · External Since: Feb 26, 2007 Posts: 21

Quoting Marcus Better <marcus DeleteThis @better.se>:

> Tom Marble wrote:
>> Packaging
>> ---------
>
> I think we need to add:
>
> * Library ABI handling and support for parallel installation of multiple
> versions (as discussed in another thread [1]).
>
> Marcus
>
> [1] http://thread.gmane.org/gmane.linux.debian.devel.java/6537

Thanks for mentioning it. I second that notion.

manfred

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

manfred · External Since: Feb 26, 2007 Posts: 21

Quoting Petteri RÃ¤ty <betelgeuse DeleteThis @gentoo.org>:

> manfred DeleteThis @mosabuam.com kirjoitti:
>> Quoting Marcus Better <marcus DeleteThis @better.se>:
>>
>>> Tom Marble wrote:
>>>> Packaging
>>>> ---------
>>>
>>> I think we need to add:
>>>
>>> * Library ABI handling and support for parallel installation of multiple
>>> versions (as discussed in another thread [1]).
>>>
>>> Marcus
>>>
>>> [1] http://thread.gmane.org/gmane.linux.debian.devel.java/6537
>>
>> Thanks for mentioning it. I second that notion.
>>
>> manfred
>>
>>
>
> We solved this by installing to /usr/share/<pkg>-<slot> where slot is
> roughly the ABI (we increase it every time a new version breaks
> something using the library).

Very nice. So you actually allow a package to be installed in various
version numbers in parallel? How do you handle man pages and all sorts
of stuff like that (e.g. executable in /usr/bin has version number in
command?)

manfred

manfred · External Since: Feb 26, 2007 Posts: 21

Quoting Petteri RÃ¤ty <betelgeuse RemoveThis @gentoo.org>:

>>> We solved this by installing to /usr/share/<pkg>-<slot> where slot is
>>> roughly the ABI (we increase it every time a new version breaks
>>> something using the library).
>>
>> Very nice. So you actually allow a package to be installed in various
>> version numbers in parallel? How do you handle man pages and all sorts
>> of stuff like that (e.g. executable in /usr/bin has version number in
>> command?)
>>
>> manfred
>
> It's not mandatory to install man pages on Gentoo but of course helpful
> for users. The thing is that besides the JDK itself I haven't see Java
> applications shipping man pages so far (reasons are easy to guess). As
> for the executable versioning, yes we version the executables if it's
> useful to install multiple versions at the same time. Most of the time
> it's only prudent to keep the executable in the latest version and
> libraries for the old versions. For example if the different versions
> use the same subdirectory in /home then it would be dangerous to let the
> user mix versions like that but in the case of for example Netbeans it's
> safe as the settings directory is versioned so there is for example:
> /usr/bin/netbeans-5.5

Cool... sounds very reasonable. Something like that should become part
of the Debian Java Policy IMHO..

manfred

Paul Cager · External Since: Dec 16, 2006 Posts: 6

Petteri Räty wrote:
> manfred.RemoveThis@mosabuam.com kirjoitti:
>> Quoting Marcus Better <marcus.RemoveThis@better.se>:
>>
>>> Tom Marble wrote:
>>>> Packaging
>>>> ---------
>>> I think we need to add:
>>>
>>> * Library ABI handling and support for parallel installation of multiple
>>> versions (as discussed in another thread [1]).
>>>
>>> Marcus
>>>
>>> [1] http://thread.gmane.org/gmane.linux.debian.devel.java/6537
>> Thanks for mentioning it. I second that notion.
>>
>> manfred
>>
>>
>
> We solved this by installing to /usr/share/<pkg>-<slot> where slot is
> roughly the ABI (we increase it every time a new version breaks
> something using the library).

That's interesting - a compromise between keeping *every* version of the
Jar, and just the latest.

A couple of questions spring to mind:

* How do you detect when a new version breaks the ABI? It seems quite
complicated. Do you use a tool to compare the classes / method
signatures, etc, or do you only bump the slot number if an application
fails?

* Supposing I (as an end-user) download an application (not packaged by
gentoo) which requires version 1.5 of foo.jar; how would I translate
that to a slot number?

Thanks,
Paul

--
To UNSUBSCRIBE, email to debian-java-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Mark Wielaard · External Since: Aug 14, 2005 Posts: 13

On Sun, 2007-06-17 at 01:01 +0100, Paul Cager wrote:
> * How do you detect when a new version breaks the ABI? It seems quite
> complicated. Do you use a tool to compare the classes / method
> signatures, etc, or do you only bump the slot number if an application
> fails?

There is Japitools http://sab39.netreach.com/japi/ which you would hope
a library writer runs before releasing a new version. It gives a nice
overview of public visible changes between 2 versions of a java library.

Note that it only checks for binary compatibility, not source
compatibility though.

I don't know of any tools designed for source compatibility.

Cheers,

Mark

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Marcus Better · External Since: Nov 19, 2004 Posts: 366

Petteri Räty wrote:
> We solved this by installing to /usr/share/<pkg>-<slot> where slot is
> roughly the ABI (we increase it every time a new version breaks
> something using the library).

For a binary distribution like Debian there are other issues, for instance
*how to generate dependencies on the correct package versions when building
packages,
*how to make sure that dependents use the correct library version at
run-time.

I think we will need something similar to the shlibs system, no?

Marcus

--
To UNSUBSCRIBE, email to debian-java-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Marcus Better · External Since: Nov 19, 2004 Posts: 366

Petteri Räty wrote:
>> I think we will need something similar to the shlibs system, no?

> Well why would they be any different for a binary distribution?

I don't know how Gentoo works, but it's not enough for us to know which ABI
version of a particular dependency we need, we also need the mapping of
that ABI to Debian packages to avoid upgrade problems.

For instance, if libfoo releases version 1.0, 2.0 and 2.1, where the ABI
changes at 2.0, then a package built against libfoo 2.1 should get a
versioned dependency on libfoo >= 2.0. That information has to be stored
somewhere, i.e., a shlibs file, so that the build process can figure it
out.

(Or am I missing something?)

Marcus

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Marcus Better · External Since: Nov 19, 2004 Posts: 366

Petteri Räty wrote:
>> I don't know how Gentoo works, but it's not enough for us to know which
>> ABI version of a particular dependency we need, we also need the mapping
>> of that ABI to Debian packages to avoid upgrade problems.
>
> So you install to /usr/share/<pkg> as you need to have the ABI in the
> package name when you don't have something like slots.

No, we can have a combination of the two. If the ABI is indicated by "slots"
(jar name, location, symlink etc) then the package name can usually stay
the same.

> I thought you can't have two versions of a package installed at the same
> time in Debian?

Correct.

> If the ABI changes between 1.0 and 2.0, they would be
> two different packages.

Not necessarily. You can either choose to keep the package name, replacing
the old version (because it's not needed), or you change the package name
so that both can be installed in parallel.

The library packaging guide [1] is useful reading, it describes many of the
issues that apply here.

Regards,

Marcus

[1] http://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html

--
To UNSUBSCRIBE, email to debian-java-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Florian Weimer · External Since: Nov 10, 2004 Posts: 607

* Tom Marble:

> 10. Filesystem conventions (FHS) for runtimes (e.g. /usr/lib/jvm)
> and libraries (e.g. /usr/share/java).

I think we also need to decide if we want to ship something under
/usr/share/java which can be ripped out and redistributed outside the
system, or if we follow the usual rules of the distribution.

For instance, many packages include a private (renamed) copy of the
ASM framework. It's generally expected that such copies are replaced
with references to a system-wide copy, to reduce the maintenance
burden (in the C land, this happens all the time with libraries such
as gettext or zlib). However, such a modified .jar file would not be
redistributable (in the technical sense).

--
To UNSUBSCRIBE, email to debian-java-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org