Hello all,
TL;DR: This is mostly of interest for people looking at/maintaining
autopkgtests and update-excuses.html or retrying failed tests.
With our current approach to running autopkgtests it sometimes happens
that a package P in -proposed gets stuck because one of its
dependencies D also gets uploaded and is broken.
Right now we always run all tests against the entirety of -proposed,
which will cause P's tests to fail due to the broken D-proposed.
However, we can do better than that: When evaluating P, we should run
P-proposed packages against D-release if possible, so that a broken
D-proposed can't cause P-proposed to be held back. Conversely, for
evaluating D-proposed all tests should be run with P-release and the
binaries of D-proposed.
This was first discussed a while ago on ubuntu-release [1], and more
recently as a bug report [2]. Now I finished the necessary
implementation in autopkgtest, britney, and the cloud worker and
rolled it out. This is essentially done by telling apt via pinning to
only install the binaries from one source that we want to update
(called the "trigger") from -proposed, and prefer everything else from
-release.
An example run can be seen for the trigger pyflakes [3], where its
reverse dependency ubuntu-dev-tools now ran [4] with python-pyflakes
1.0.0-3 from -proposed (as that is the trigger), but e. g. python-gi
3.18.0-1 from -release (although 3.18.2-2 is in -proposed). So if the
new pygobject 3.18.2-2 were broken, we could now still promote pyflakes.
Note that apt pinning is rather limited. Unfortunately you can't say
"from -proposed use these packages and any of their dependencies which
are not satisfiable in -release" -- pinning is not clever enough for
that. This means that the tests for e. g. gtk+3.0 where the -proposed
version depends on another package in -proposed cannot be run in this
isolated manner, and autopkgtest will fall back to globally enabling
-proposed. You will see something like
adt-run: WARNING: Packages src:gtk+3.0 are not installable with
per-package pinning in proposed, falling back to using all packages
from proposed:
in the log, e. g. in [5]. Ideas how to improve that are highly
appreciated!
This comes at the expense of having to do more test runs, as one
particular test run is now specific to a particular trigger. This also
means that the notion of "please re-run test foo" is now meaningless,
you always have to re-run a test against a given trigger. (Note that
this was already the case for kernel packages, but now we use this
approach for everything). In a test log, you can see the test trigger
at the top as e. g.
ADT_TEST_TRIGGERS=pyflakes/1.0.0-3
result.tar's "testinfo.json" contains the trigger in machine readable
form. I plan to expose this in a more friendly manner on
http://autopkgtest.ubuntu.com [6] soon.
Martin
[1] https://lists.ubuntu.com/archives/ubuntu-release/2015-August/003345.html
[2] https://launchpad.net/bugs/1503150
[3] http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#pyflakes
[4] https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/i386/u/ubuntuone-dev-tools/20151110_123612@/log.gz
[5] https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/i386/libr/libreoffice/20151110_131142@/log.gz
[6] https://bugs.launchpad.net/auto-package-testing/+bug/1511780
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
