Thursday 8 August 2013

Re: speeding up package builds

Matthias Klose <doko@ubuntu.com> wrote:
>I had to look at many "core" packages, which are needed for a normal
>and a
>buildd chroot, and and which are needed to build these packages for the
>chroots.
>And I was not that pleased to see how the packaging wasted CPU time. I
>think
>even for our migration from proposed to release it would make sense to
>speed up
>some builds, so that the slower architectures can catch up earlier. So
>here is
>what I found ...
>
>Parallel builds
>
>Unfortunately not enabled very often. The buildds set
>DEB_BUILD_OPTIONS="parallel=<N>", and if the build supports parallel
>builds, it
>can speed up the build substantially. Common mistakes:
>
> - debhelper based packages don't call with --parallel
> - debhelper based packages overwrite dh_auto_build and don't call
> --parallel
> - cdbs based packages don't set DEB_BUILD_PARALLEL = yes
> - Packages deep down in the stack live still in the stone age (bind9,
> db, ghostscript, ...)
>
>
>Documentation builds
>
>Many packages always build the documentation, even when only building
>the
>architecture dependent packages (that is, on our slowest buildds). The
>easy
>solution where the upstream docs are not built by default is to build
>these in
>an extra target.
>
>However I didn't find any good recipes how to write a modern debhelper
>or
>semi-modern cdbs rules file how to correctly configure a package to
>disable the
>docs build for an arch-only build, and enable the docs for an
>arch-indep build.
>Pointers welcome.
>
>
>Flavour builds
>
>There is no benefit for our proposed to release migration, however when
>testing
>a package, or building a package for the first time, I'm maybe only
>interested
>in one flavour. Disabling one flavour reduces the build time. Mostly
>seen for
>packages using different builds for building udeb's. It would be nice
>to
>disable these if DEB_STAGE=<something> is set.
>
>
>Shell-script like rules files
>
>Using a short "modern" debhelper rules file doesn't mean that you
>cannot define
>your own targets to break a configure or build target in many sub
>targets. It
>doesn't speed up the build itself, however if you debug an issue, and
>the build
>starts again ... something could be done better. Please try to at
>least
>separate the configure and build steps. Look at boost1.xy to see what
>I don't like.
>
>
>As an example, gtk+3.0 did become eight times faster by enabling
>parallel builds
>(parallel=8), disabling the doc build, and the udeb flavour. Just
>picking this
>up, because I had to build it several times.
>
>Hope this helps a bit in making package builds and migrations a bit
>faster.

The last python-qt4 sync I did from Debian took over a day to migrate due to the amount of time the autopkgtests took. It was at least twice as long as the build time on the slowest arch. Is there work planned to speed up these tests?

Scott K


--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel