On Sun, May 11, 2014 at 11:08:31PM +0100, Colin Watson wrote:
> [This is a Foundations/CI project; I haven't posted a status update on
> it before, but a few people have asked about it in private mail, so it
> might be helpful to have a public update to refer to.]
>
> We've been working on reorganising the infrastructure for the live
> filesystem parts of our image builds. At the moment there's a single
> dedicated builder for each architecture that the central image build
> system (lp:ubuntu-cdimage, running on nusakan.canonical.com) can ask to
> produce various live filesystems; ubuntu-cdimage then fetches the
> results from it, performs some final polishing steps (for ISO images,
> this involves building a pool of .debs among other things, so is a
> fairly complex procedure requiring debian-cd and a local mirror; for
> phone images it's relatively simple), and publishes the output on
> cdimage.ubuntu.com. In the case of phone images, system-image then
> steps in to turn these into the system-image format and compute deltas
> etc.
>
> This is serviceable for what we're doing with it right now, but there
> are a few motivations for improving things which have been on the
> wishlist for some time:
>
> * It's not at all horizontally scalable. We notice this most around
> milestone releases when we try to do lots of image builds at once,
> and we can only do one at a time per architecture. This is basically
> why respin cycles for milestones take eight hours or whatever
> ridiculous amount it is.
>
> * We can't sensibly extend this to experimental builds.
>
> * To a first approximation, ubuntu-cdimage has only been deployed once
> (although I rewrote it a while back, and one of the goals of the
> rewrite was to make deployment easier).
>
> We discussed this at the release engineering sprint last year, and the
> general consensus was that we should move live filesystem building into
> Launchpad. This has a number of advantages: both build farms are
> normally well within capacity except for occasional bursts that tend to
> be at complementary times; we would be able to run many more image
> builds at once and thus drastically reduce the time required for a full
> respin; it would be much easier to monitor builds in progress; builds
> will be cancellable without sysadmin intervention; and we'll get one or
> two extra builders on most architectures by merging the build farms, at
> least where the hardware in use is new enough to make that worthwhile.
>
> I've been working on this on and off for a while, and essentially
> full-time since 14.04 released. The current status is as follows:
>
> * The launchpad-buildd package in production already has most of the
> code required to build live filesystems; this was done towards the
> end of last year. More recent testing uncovered a few small but
> critical problems with this code with various real-world image types
> (buggy handling of symlinks and of duplicate files in build output);
> one fix is in trunk and the other is awaiting review.
>
> * The bulk of the required changes to Launchpad itself have been
> through one review pass and are awaiting re-review. The required
> extensions to the web UI are still only on my local disk, as I have a
> few more tests to write first, but I expect to finish that in the
> first half of the coming week.
>
> * I have a local installation that is capable of building full live
> filesystem images, tested with both Ubuntu Desktop and Ubuntu Touch
> (i386).
>
> * I have a local ubuntu-cdimage branch that can request live filesystem
> builds from a local Launchpad instance and wait for them to complete.
> It cannot yet download the results.
>
> There's still a fair bit to do, but my aim is to have this landed and in
> production in time for the core sprint in Malta in the last week of May.
>
> The engineers developing the CI Airline need to be able to build live
> filesystems against PPAs as a component of their work, and so I've been
> in discussions with them to make sure that we will be able to use the
> same software for this, and where possible the same infrastructure,
> rather than duplicating work. In anticipation of this requirement, the
> database modelling for live filesystems in Launchpad includes the
> facility to associate a build with a PPA, although the pieces required
> to pass that through to the sources.list used to construct the live
> filesystem aren't quite in place yet. The CI Airline will need its own
> cut-down ubuntu-cdimage and system-image instances, and I'll be working
> with Paul Larson early this week to get the ubuntu-cdimage part of that
> up and running; phone image builds are simple enough from this point of
> view that I don't expect that to be difficult. Although I can't speak
> for system-image, I think we are well on course to have all the rest of
> this either done before Malta or easy to finish off there.
I'm mostly off until Malta, only around this Friday and Monday-Tuesday
of next week (16, 19 and 20th of May) so if you have cdimage setup by
then, it's possible I can help setting things up for system-image
builds, if you need more time, then it's just a matter of sitting
together with Paul in Malta and getting the remaining bits sorted out (I
expect we'll need some minor tweaks to the code but nothing too scary).
>
> Cheers,
>
> --
> Colin Watson [cjwatson@ubuntu.com]
>
> --
> Mailing list: https://launchpad.net/~ubuntu-phone
> Post to : ubuntu-phone@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~ubuntu-phone
> More help : https://help.launchpad.net/ListHelp
--
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com
