Java buildd service

Michael Koch · External Since: Feb 18, 2005 Posts: 574

Hello list,

I have setup a local buildd locally to be able to testbuild all packages
maintained by the Debian Java Maintainers. In the first test builds only
the usual suspects failed which have already filed FTBFS bugs in the
BTS.

I would like to provide this service to be able to test all the java
packages when a dependency gets updated and may be API incompatible.
This allows us to find issues before we upload the package into the
official archive. We can also test new toolchains.

The setup currently uses pbuilder but I will migrate it to
cowbuilder/lvmpbuilder soon to get a better performance.

If you packages you want me to test build with all its
reverse-dependencies please mail me.

What do you think?

Cheers,
Michael
--
.''`. | Michael Koch <konqueror RemoveThis @gmx.de>
: :' : | Free Java Developer <http://www.classpath.org>
`. `' |
`- | 1024D/BAC5 4B28 D436 95E6 F2E0 BD11 5923 A008 2763 483B

--
To UNSUBSCRIBE, email to debian-java-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Marcus Better · External Since: Nov 19, 2004 Posts: 393

Michael Koch wrote:
> the usual suspects failed which have already filed FTBFS bugs in the
> BTS.

What was the result for dom4j? I would like to have more data on #427456.
What is the arch and CPU of the build system?

Regards,

Marcus

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Michael Koch · External Since: Feb 18, 2005 Posts: 574

On Mon, Jul 02, 2007 at 10:53:17AM +0200, Marcus Better wrote:
> Michael Koch wrote:
> > the usual suspects failed which have already filed FTBFS bugs in the
> > BTS.
>
> What was the result for dom4j? I would like to have more data on #427456.
> What is the arch and CPU of the build system?

The buildf failed but differently:

[junit] Running org.dom4j.xpath.MatrixConcatTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.833 sec

BUILD FAILED
/tmp/buildd/dom4j-1.6.1+dfsg/build.xml:276: Test org.dom4j.xpath.MatrixConcatTest failed

I think it would be okay for now to turn off the checks during build-time to fix #427456.

The buildd currently runs on i386 userspace on amd64 cpu. I will add an
additional flavor to be able to build on i386 *and* amd64. I have a fast
enough powerpc machine here too which needs some love and can then act
as another buildd too.

Cheers,
Michael
--
.''`. | Michael Koch <konqueror DeleteThis @gmx.de>
: :' : | Free Java Developer <http://www.classpath.org>
`. `' |
`- | 1024D/BAC5 4B28 D436 95E6 F2E0 BD11 5923 A008 2763 483Bd

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Marcus Better · External Since: Nov 19, 2004 Posts: 393

Michael Koch wrote:
> The buildf failed but differently:
>
>
> [junit] Running org.dom4j.xpath.MatrixConcatTest
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.833 sec

Ok, so the ThreadingTest succeeded at least.

Could you send me the test result for the MatrixConcatTest from
build/test-results/xml, it should contain the exception message.

> I think it would be okay for now to turn off the checks during build-time
> to fix #427456.

Perhaps, but I don't like to turn off test suites just because they fail Smile

It can make multi-threaded programs break randomly...

> The buildd currently runs on i386 userspace on amd64 cpu.

Is it an SMP system?

Marcus

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Michael Koch · External Since: Feb 18, 2005 Posts: 574

On Mon, Jul 02, 2007 at 01:36:19PM +0200, Marcus Better wrote:
> Michael Koch wrote:
> > The buildf failed but differently:
> >
> >
> > [junit] Running org.dom4j.xpath.MatrixConcatTest
> > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.833 sec
>
> Ok, so the ThreadingTest succeeded at least.
>
> Could you send me the test result for the MatrixConcatTest from
> build/test-results/xml, it should contain the exception message.

<testcase classname="org.dom4j.xpath.MatrixConcatTest" name="testMatrixConcat" time="0.458">
<error message="Exception occurred evaluting XPath: matrix-concat('EQUITY_',/product/cashflows/CashFlow/XREF). Exception: No Such Function matrix-concat" type="org.dom4j.XPathException">org.dom4j.XPathException: Exception occurred evaluting XPath: matrix-concat('EQUITY_',/product/cashflows/CashFlow/XREF). Exception: No Such Function matrix-concat
at org.dom4j.xpath.DefaultXPath.handleJaxenException(DefaultXPath.java:374)
at org.dom4j.xpath.DefaultXPath.selectNodes(DefaultXPath.java:134)
at org.dom4j.tree.AbstractNode.selectNodes(AbstractNode.java:166)
at org.dom4j.xpath.MatrixConcatTest.testMatrixConcat(MatrixConcatTest.java:47)
at org.dom4j.xpath.MatrixConcatTest.testMatrixConcat(MatrixConcatTest.java:36)
</error>
</testcase>

Doesnt make much sense to me currently, but I havnt looked into the code.
>
> > I think it would be okay for now to turn off the checks during build-time
> > to fix #427456.
>
> Perhaps, but I don't like to turn off test suites just because they fail Smile

> It can make multi-threaded programs break randomly...

I dont like that idea either.

> > The buildd currently runs on i386 userspace on amd64 cpu.
>
> Is it an SMP system?

This is a single-cpu/single-core machine.

Cheers,
Michael
--
.''`. | Michael Koch <konqueror DeleteThis @gmx.de>
: :' : | Free Java Developer <http://www.classpath.org>
`. `' |
`- | 1024D/BAC5 4B28 D436 95E6 F2E0 BD11 5923 A008 2763 483B

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Marcus Better · External Since: Nov 19, 2004 Posts: 393

Michael Koch wrote:
> This is a single-cpu/single-core machine.

That fits the pattern. The problem seems to hit SMP machines.

Marcus

--
To UNSUBSCRIBE, email to debian-java-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Paul Cager · External Since: Nov 30, 2006 Posts: 123

Michael Koch wrote:
> Hello list,
>
>
> I have setup a local buildd locally to be able to testbuild all packages
> maintained by the Debian Java Maintainers. In the first test builds only
> the usual suspects failed which have already filed FTBFS bugs in the
> BTS.
>
> I would like to provide this service to be able to test all the java
> packages when a dependency gets updated and may be API incompatible.
> This allows us to find issues before we upload the package into the
> official archive. We can also test new toolchains.
>
> The setup currently uses pbuilder but I will migrate it to
> cowbuilder/lvmpbuilder soon to get a better performance.
>
> If you packages you want me to test build with all its
> reverse-dependencies please mail me.
>
> What do you think?
>
>
> Cheers,
> Michael

Michael,

Is it possible to set up the build environment so that access to the
network will fail (after downloading the Build-depends of course)? I'm
thinking of packages that attempt to download binary blobs (e.g.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=424466).

Thanks,
Paul

--
To UNSUBSCRIBE, email to debian-java-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Michael Koch · External Since: Feb 18, 2005 Posts: 574

On Tue, Jul 10, 2007 at 09:46:32AM +0100, Paul Cager wrote:
> Is it possible to set up the build environment so that access to the
> network will fail (after downloading the Build-depends of course)? I'm
> thinking of packages that attempt to download binary blobs (e.g.
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=424466).

No, pbuilder uses apt to download build dependencies that are not in its
cache. My special setup even updates the apt lists always to allow build
ing with just build other packages (build-dependencies).

I have thought about this problem and I think the we should have a
preloadable library (packaged as deb) that prevents all network
operations. This should be easy to do with re-implementing the socket()
function from glibc. Then its still possible to ioctls into the kernel
to do create a socket but thats unlikely occur. Then something like

nonet debuild ....

should be possible to build without network access without breaking your
system. This can then be built into our buildd.

I'm investigating this solution more.

Cheers,
Michael
--
.''`. | Michael Koch <konqueror.TakeThisOut@gmx.de>
: :' : | Free Java Developer <http://www.classpath.org>
`. `' |
`- | 1024D/BAC5 4B28 D436 95E6 F2E0 BD11 5923 A008 2763 483B

--
To UNSUBSCRIBE, email to debian-java-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Paul Wise · External Since: Mar 27, 2007 Posts: 33

On 7/2/07, Michael Koch <konqueror.DeleteThis@gmx.de> wrote:

> I have setup a local buildd locally to be able to testbuild all packages
> maintained by the Debian Java Maintainers. In the first test builds only
> the usual suspects failed which have already filed FTBFS bugs in the
> BTS.

I suggest contacting Lucas Nussbaum (of debian-qa), who has been
rebuilding the archive regularly on a 5000 node distributed cluster in
France (no net access from inside the cluster). I'm sure he would be
more than willing to help setup whatever special situation is needed
for debian-java. He did a presentation about this at debconf, video is
probably on meetings-archive.d.n by now. I imagine the tests you
mention across all of debian-java would take a matter of minutes.

Latest FTBFS logs from his archive rebuild are here:

http://people.debian.org/~lucas/logs/2007/07/08/

--
bye,
pabs

http://wiki.debian.org/PaulWise

--
To UNSUBSCRIBE, email to debian-java-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Michael Koch · External Since: Feb 18, 2005 Posts: 574

On Wed, Jul 11, 2007 at 09:55:22AM +1000, Paul Wise wrote:
> On 7/2/07, Michael Koch <konqueror DeleteThis @gmx.de> wrote:
>
> >I have setup a local buildd locally to be able to testbuild all packages
> >maintained by the Debian Java Maintainers. In the first test builds only
> >the usual suspects failed which have already filed FTBFS bugs in the
> >BTS.
>
> I suggest contacting Lucas Nussbaum (of debian-qa), who has been
> rebuilding the archive regularly on a 5000 node distributed cluster in
> France (no net access from inside the cluster). I'm sure he would be
> more than willing to help setup whatever special situation is needed
> for debian-java. He did a presentation about this at debconf, video is
> probably on meetings-archive.d.n by now. I imagine the tests you
> mention across all of debian-java would take a matter of minutes.
>
> Latest FTBFS logs from his archive rebuild are here:
>
> http://people.debian.org/~lucas/logs/2007/07/08/

I've spoken with Lucas on IRC before and his work is really useful. But
his work has a different goal. Lucas does checks on current archive.
My buildd can do builds before packages enter the archive. E.g. I've
used it to rebuild all packages build-depending on ant before I uploaded
ant 1.7.0 into the archive to make sure it dont break too much. This way
I found a breakage in ow-util-ant-tasks with the new ant version.

Cheers,
Michael
--
.''`. | Michael Koch <konqueror DeleteThis @gmx.de>
: :' : | Free Java Developer <http://www.classpath.org>
`. `' |
`- | 1024D/BAC5 4B28 D436 95E6 F2E0 BD11 5923 A008 2763 483B

--
To UNSUBSCRIBE, email to debian-java-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Paul Wise · External Since: Mar 27, 2007 Posts: 33

On 7/11/07, Michael Koch <konqueror.DeleteThis@gmx.de> wrote:

> My buildd can do builds before packages enter the archive. E.g. I've
> used it to rebuild all packages build-depending on ant before I uploaded
> ant 1.7.0 into the archive to make sure it dont break too much. This way
> I found a breakage in ow-util-ant-tasks with the new ant version.

Hmm, I thought that would be possible with the cluster too. Anyways.

--
bye,
pabs

http://wiki.debian.org/PaulWise

--
To UNSUBSCRIBE, email to debian-java-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org