Getting Maven Into Main - What Next?

Paul Cager · External Since: Nov 30, 2006 Posts: 123

Now that we are making progress with packaging Maven's
Build-Dependencies, I'm beginning to wonder what happens next. I've
noticed from the JavaCC packaging that it seems to be all right to
include a "bootstrap jar" in the upstream source. Is this right? I guess
to be valid:

* When building the package a new Jar must be built (from source)
that could replace the bootstrap Jar.

* There must be no licensing issues around distributing the bootstrap
Jar.

If so then should we create the Maven package by bootstrapping?

* Maven build-depends on wagon, plexus, doxia etc.

* Create a local repository for Maven, e.g. debian/.m2/repository.
For each build-dependency create the appropriate directory structure
(e.g. debian/.m2/repository/plexus/plexus-utils/1.0.2/).

* Copy (or symlink) the build-depends jars from /usr/share/java to
our repository. This should ensure that maven does not try to download
any dependencies. (I imagine maven also has an option to inhibit
downloads; it would be safest to set this, and/or set http_proxy).

* Set "localRepository" to "debian/.m2/repository", and call maven's
bootstrap shell script. (this process is described in
http://maven.apache.org/guides/development/guide-building-m2.html)

Any comments?

--
To UNSUBSCRIBE, email to debian-java-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Manfred Moser · External Since: Mar 04, 2007 Posts: 9

On Friday May 25 2007, Paul Cager wrote:
> Now that we are making progress with packaging Maven's
...

Hi Paul and all,

Just fyi I am an active Maven and long time Debian/Ubuntu user with
very little packaging know how....

First I want to recommend two resources (both free)

The books

Better Builds with Maven
http://www.mergere.com/m2book_download.jsp

Maven: The Definitive Guide (1.0 Alpha 1)
http://www.sonatype.com/book/index.html

> * Create a local repository for Maven, e.g.
> debian/.m2/repository. For each build-dependency create the
> appropriate directory structure (e.g.
> debian/.m2/repository/plexus/plexus-utils/1.0.2/).

I would actually suggest to start building a debian build server
internal maven repository.

http://maven.apache.org/guides/introduction/introduction-to-repositories.html

Together with a proxy (e.g. proximity .. also documented in the books)
it can be ensured that a build is basically done without going out to
the internet for any artifacts and it is guaranteed that only allowed
internal artifacts are use.

The repository should only contain packages that are in debian. Maven
could actually deploy artifacts to the repo as part of the build
process.

For other builds it would be good if the packages can deploy to the
repo as well. Than any maven build can just declare the right debian
dependency.

> * Copy (or symlink) the build-depends jars from /usr/share/java
> to our repository. This should ensure that maven does not try to
> download any dependencies. (I imagine maven also has an option to
> inhibit downloads; it would be safest to set this, and/or set
> http_proxy).

Use of proxies is a bit documented here
http://maven.apache.org/guides/mini/guide-proxies.html

To run maven in offline mode there is a -o flag.

> * Set "localRepository" to "debian/.m2/repository", and call
> maven's bootstrap shell script. (this process is described in
> http://maven.apache.org/guides/development/guide-building-m2.html)

It might be a good idea to create a "company" pom for Debian that
contains all settings and so on that apply to all Debian packages
that are build with Maven. Otherwise a debian profile in a
settings.xml could achieve similar things.

I am sure the maven team could be of great help. I told Jason at Java
One that the Debian team is packaging Maven and he was astonished and
delighted. He or somebody else might be able to give a good tip here
and there.

I hope this helps a bit.

manfred

--
Manfred Moser
http://www.mosabuam.com
skype: mosabua
http://www.linkedin.com/in/manfredmoser

--
To UNSUBSCRIBE, email to debian-java-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Paul Cager · External Since: Nov 30, 2006 Posts: 123

Manfred Moser wrote:
> [... snip some very useful information ...]
>
> I hope this helps a bit.

Yes, it certainly does, and thank you. I'll have a good read of it later
today when I'm more awake!

>
> manfred
>

Thanks,
Paul

--
To UNSUBSCRIBE, email to debian-java-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Paul Cager · External Since: Nov 30, 2006 Posts: 123

Paul Cager wrote:
> This is the "desert island test" [1],

No it isn't. The Desert Island Test is something quite different, to do
with the freeness of the upstream license. Please ignore this bit, as I
hadn't got my brain in gear!

The Debian prohibition against downloading during a package build
stands, though.

> [1] http://people.debian.org/~bap/dfsg-faq.html

--
To UNSUBSCRIBE, email to debian-java-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Paul Cager · External Since: Nov 30, 2006 Posts: 123

Manfred Moser wrote:
> On Friday May 25 2007, Paul Cager wrote:
>> Now that we are making progress with packaging Maven's
> ..
>
> Hi Paul and all,
>
> Just fyi I am an active Maven and long time Debian/Ubuntu user with
> very little packaging know how....

I have practically no experience with Maven (yet!), so it's good to meet
someone who is active with it. Would you be interested in getting
involved with the testing of any packaging we produce? We'd do some
basic testing, of course, but it would be nice if there was testing
against "real world" projects.

Just to summarise what I understand to be plans for Maven:

Firstly we want to package maven itself, so that users can "aptitude
install maven", and Maven will work just as though the users had
downloaded and installed it manually. Secondly we want to be able to use
Maven within the Debian packaging system to package upstream projects
that use Maven.

At the moment I am concentrating on the first objective only.

>
> First I want to recommend two resources (both free)
>
> The books
>
> Better Builds with Maven
> http://www.mergere.com/m2book_download.jsp
>
> Maven: The Definitive Guide (1.0 Alpha 1)
> http://www.sonatype.com/book/index.html

Great! Once the children are in bed I'll get reading.

>> * Create a local repository for Maven, e.g.
>> debian/.m2/repository. For each build-dependency create the
>> appropriate directory structure (e.g.
>> debian/.m2/repository/plexus/plexus-utils/1.0.2/).
>
> I would actually suggest to start building a debian build server
> internal maven repository.

Debian imposes quite strict (but sensible) rules on how its packages can
be built. One restriction is that the package must be able to be built
from source, without downloading anything (except other packages from
the Debian archive). This is the "desert island test" [1], and probably
disallows access to a standard maven repository. This is why I am
proposing creating a temporary local repository (on the build machine)
each time a package is built.

> http://maven.apache.org/guides/introduction/introduction-to-repositories.html
> The repository should only contain packages that are in debian. Maven
> could actually deploy artifacts to the repo as part of the build
> process.

Yes, once we have Maven available as a Debian package, we will have to
look at this sort of thing.

> To run maven in offline mode there is a -o flag.

Thanks - that's one we will need.

> It might be a good idea to create a "company" pom for Debian that
> contains all settings and so on that apply to all Debian packages
> that are build with Maven. Otherwise a debian profile in a
> settings.xml could achieve similar things.

Notes, thanks.

>
> I am sure the maven team could be of great help. I told Jason at Java
> One that the Debian team is packaging Maven and he was astonished and
> delighted. He or somebody else might be able to give a good tip here
> and there.
>
> I hope this helps a bit.
>
> manfred
>

Thanks for your help. It is much appreciated.

Paul

[1] http://people.debian.org/~bap/dfsg-faq.html

--
To UNSUBSCRIBE, email to debian-java-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Florian Weimer · External Since: Nov 10, 2004 Posts: 648

* Paul Cager:

> Now that we are making progress with packaging Maven's
> Build-Dependencies, I'm beginning to wonder what happens next. I've
> noticed from the JavaCC packaging that it seems to be all right to
> include a "bootstrap jar" in the upstream source. Is this right? I guess
> to be valid:
>
> * When building the package a new Jar must be built (from source)
> that could replace the bootstrap Jar.

This is a bit tricky. Theoretically, you are supposed to bootstrap
from an installed version (IOW, a self-depdency). However, several
packages (Ocaml, for instance) bootstrap from blobs. In case of
source code generators, it's also common to ship the processed source
files.

--
To UNSUBSCRIBE, email to debian-java-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Trygve_Laugstøl · External Since: Mar 05, 2007 Posts: 8

Paul Cager wrote:
> Now that we are making progress with packaging Maven's
> Build-Dependencies, I'm beginning to wonder what happens next. I've
> noticed from the JavaCC packaging that it seems to be all right to
> include a "bootstrap jar" in the upstream source. Is this right? I guess
> to be valid:
>
> * When building the package a new Jar must be built (from source)
> that could replace the bootstrap Jar.

Makes sense

> * There must be no licensing issues around distributing the bootstrap
> Jar.
>
>
> If so then should we create the Maven package by bootstrapping?
>
> * Maven build-depends on wagon, plexus, doxia etc.
>
> * Create a local repository for Maven, e.g. debian/.m2/repository.
> For each build-dependency create the appropriate directory structure
> (e.g. debian/.m2/repository/plexus/plexus-utils/1.0.2/).

Use the install:install-file goal to properly install a file into a
repository.

> * Copy (or symlink) the build-depends jars from /usr/share/java to
> our repository. This should ensure that maven does not try to download
> any dependencies. (I imagine maven also has an option to inhibit
> downloads; it would be safest to set this, and/or set http_proxy).

It will not download stuff, but to be sure run with the -o flag.

I wonder if it would be feasible to make a LD preload thing that would
prevent all forms of external network communication that could be use to
make sure that the build isn't fetching stuff from the internet?

> * Set "localRepository" to "debian/.m2/repository", and call maven's
> bootstrap shell script. (this process is described in
> http://maven.apache.org/guides/development/guide-building-m2.html)
>
> Any comments?

I would just package all the binaries into a repository inside a tarball
and the Maven 2 source code and built it with Ant. Once you have a Maven
2 package you can start building all it's build dependencies and then
finally rebuild the Maven 2 "proper" again.

I will most likely be on IRC tonight if anyone want to chat on the
practical parts of building Maven.

--
Trygve

--
To UNSUBSCRIBE, email to debian-java-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Trygve_Laugstøl · External Since: Mar 05, 2007 Posts: 8

Florian Weimer wrote:
> * Paul Cager:
>
>> Now that we are making progress with packaging Maven's
>> Build-Dependencies, I'm beginning to wonder what happens next. I've
>> noticed from the JavaCC packaging that it seems to be all right to
>> include a "bootstrap jar" in the upstream source. Is this right? I guess
>> to be valid:
>>
>> * When building the package a new Jar must be built (from source)
>> that could replace the bootstrap Jar.
>
> This is a bit tricky. Theoretically, you are supposed to bootstrap
> from an installed version (IOW, a self-depdency). However, several
> packages (Ocaml, for instance) bootstrap from blobs. In case of
> source code generators, it's also common to ship the processed source
> files.

If someone would say that if this was allowed with Maven it would make
life so much easier as Maven has to many build dependencies. Then
Michael Koch's binary package could be used to build all of those
dependencies and then a "proper" Maven could be built easily.

--
Trygve

--
To UNSUBSCRIBE, email to debian-java-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Paul Cager · External Since: Nov 30, 2006 Posts: 123

Florian Weimer wrote:
> * Paul Cager:
>
>> Now that we are making progress with packaging Maven's
>> Build-Dependencies, I'm beginning to wonder what happens next. I've
>> noticed from the JavaCC packaging that it seems to be all right to
>> include a "bootstrap jar" in the upstream source. Is this right? I guess
>> to be valid:
>>
>> * When building the package a new Jar must be built (from source)
>> that could replace the bootstrap Jar.
>
> This is a bit tricky. Theoretically, you are supposed to bootstrap
> from an installed version (IOW, a self-depdency).

So we couldn't use bootstrapping to get the *first* version into Debian?
We could only use version (n-1) to build version (n)?

> However, several
> packages (Ocaml, for instance) bootstrap from blobs.

Ah - so we might get away with it? Maybe I should use ask the FTP masters?

> In case of
> source code generators, it's also common to ship the processed source
> files.

Yes, that's a possibility.

--
To UNSUBSCRIBE, email to debian-java-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Florian Weimer · External Since: Nov 10, 2004 Posts: 648

* Paul Cager:

>> This is a bit tricky. Theoretically, you are supposed to bootstrap
>> from an installed version (IOW, a self-depdency).
>
> So we couldn't use bootstrapping to get the *first* version into Debian?

If there are architecture-dependent binary packages, they need to
built and uploaded manually, for each architecture. The same thing
happens when you break the chain. The upload history for MLton might
be quite instructive here.

It's basically a non-issue for architecture-independent packages. The
only requirement is that the package can be rebuilt with itself.

> We could only use version (n-1) to build version (n)?

You need to start from somewhere, obviously. Cool

>> However, several
>> packages (Ocaml, for instance) bootstrap from blobs.
>
> Ah - so we might get away with it? Maybe I should use ask the FTP
> masters?

The current Java policy requires that you strip the JARs. Not that it
is rigorously followed, but you should decide first what you actually
want, and then you can pester the ftpmasters.

--
To UNSUBSCRIBE, email to debian-java-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Paul Cager · External Since: Nov 30, 2006 Posts: 123

Florian Weimer wrote:
> * Paul Cager:
>
>> Now that we are making progress with packaging Maven's
>> Build-Dependencies, I'm beginning to wonder what happens next. I've
>> noticed from the JavaCC packaging that it seems to be all right to
>> include a "bootstrap jar" in the upstream source. Is this right? I guess
>> to be valid:
>>
>> * When building the package a new Jar must be built (from source)
>> that could replace the bootstrap Jar.
>
> This is a bit tricky. Theoretically, you are supposed to bootstrap
> from an installed version (IOW, a self-depdency). However, several
> packages (Ocaml, for instance) bootstrap from blobs. In case of
> source code generators, it's also common to ship the processed source
> files.

Thinking about it, am I worrying unnecessarily here? The source
downloaded from svn *does* include Jars, but I think these are only used
to run the unit tests. (I'm guessing; Trygve could you confirm this,
please?).

Once we have the build-dependencies packaged we should be able to
download the svn source, delete the jars and create an upstream source
tarball. Then debian/rules could just call the bootstrap.sh script
(possibly amended to prevent net access).

I don't think my brain is working this week...

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Manfred Moser · External Since: Mar 04, 2007 Posts: 9

On Friday May 25 2007, Paul Cager wrote:

> I have practically no experience with Maven (yet!), so it's good to
> meet someone who is active with it. Would you be interested in
> getting involved with the testing of any packaging we produce? We'd
> do some basic testing, of course, but it would be nice if there was
> testing against "real world" projects.

Yeah. For sure. I could at least build it against my project (large
JEE project with lots of subprojects using all sorts of plugins). I
am currently using the upstream version as well as Michael Kochs
package. Both are fine.

> Just to summarise what I understand to be plans for Maven:
>
> Firstly we want to package maven itself, so that users can
> "aptitude install maven",

aptitude install maven2
I assume...

> and Maven will work just as though the
> users had downloaded and installed it manually.

> >> * Create a local repository for Maven, e.g.
> >> debian/.m2/repository. For each build-dependency create the
> >> appropriate directory structure (e.g.
> >> debian/.m2/repository/plexus/plexus-utils/1.0.2/).
> >
> > I would actually suggest to start building a debian build server
> > internal maven repository.
>
> Debian imposes quite strict (but sensible) rules on how its
> packages can be built. One restriction is that the package must be
> able to be built from source, without downloading anything (except
> other packages from the Debian archive). This is the "desert island
> test" [1], and probably disallows access to a standard maven
> repository. This is why I am proposing creating a temporary local
> repository (on the build machine) each time a package is built.

I understand that restriction. I was actually proposing to create a
debian private repo as part of the Debian archive into which all
maven packages and dependencies go as well as into which all future
packages can get their artifacts into and use others from there.

I look forward to the progress. I wish I could help more but time is
very short at the moment... at least I hope to be able to help by
giving my input.

manfred
--
Manfred Moser
http://www.mosabuam.com
skype: mosabua
http://www.linkedin.com/in/manfredmoser

--
To UNSUBSCRIBE, email to debian-java-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Marcus Better · External Since: Nov 19, 2004 Posts: 393

Florian Weimer wrote:
>> * When building the package a new Jar must be built (from source)
>> that could replace the bootstrap Jar.
>
> This is a bit tricky. Theoretically, you are supposed to bootstrap
> from an installed version (IOW, a self-depdency).

I assume maven2 will be an architecture-independent package, so IIUC this
should work:
1. Build the first version with any Maven install, such as Michael's binary
package.
2. Install.
3. Let maven2 build-dep on itself, rebuild and install.
3. Upload!

Or am I missing something?

Marcus

--
To UNSUBSCRIBE, email to debian-java-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org