Monday 5 December 2022

Re: +1 maintenance report

Sorry to hear that you had a frustrating time with +1 maintenance last week.
Large transitions can definitely be frustrating because many times, it's so
unclear how to make progress on them.

Some notes:

- The request.cgi API endpoint does deduplication of test requests on the
backend. So provided you are generating your list of tests with matching
logic for triggers and options (--all-proposed vs not), it's safe to
blindly rerun retry-autopkgtest-regressions as many times as you need to
without worrying about duplicates driving up the build queue times.

- Although you didn't feel that you made much progress, it is still very
valuable in situations like this to have somebody monitoring the things
that have continued to fail and drill into them as necessary. So I would
not consider this time wasted!

If my suggestion had been more than an IRC driveby on the weekend due to my
limited time, I would have mentioned the specific pipeline I was using to
retrigger tests and why:

$ retry-autopkgtest-regressions --all-proposed | grep '=perl%2F' \
| xargs -rn1 -P10 wget --load-cookies ~/.cache/autopkgtest.cookie -O-

First, this skips the normal '--blocks perl' option to
retry-autopkgtest-regressions in favor of a postprocessing grep. The reason
for this is that using --blocks doesn't just limit the tests reported, it
also limits the triggers emitted. So if there is a package, say,
libcatmandu-mab2-perl, which is failing its autopkgtests against perl
because it's using the release version of libcatmandu-mab2-perl, *and*
failing its autopkgtests against itself because it's using the release
version of perl, this picks up both triggers; which not only ensures the
package gets tested with both of these packages from -proposed, it also
credits any test pass against both packages in update_excuses.

Second, even though the above should already pick up more packages from
-proposed that we need, I've also added --all-proposed, saying to grab all
packages from -proposed. Two reasons for this:

- As previously discussed, there were select packages in the autopkgtest
base images that were newer than the versions in lunar release, due to
building against kinetic-security. For a mass-retry, --all-proposed was
more likely to succeed than to cause false negatives.

- In some cases, a test might require two packages (both perl and
libfoo-perl) from -proposed in order to pass, but for whatever reason the
test against libfoo-perl had already passed (because of luck on the
autopkgtest -proposed fallback handling, or a manual retry done in
isolation without considering perl itself, or $other), while the same
test against perl had failed. In that case,
retry-autopkgtest-regressions would output a line with only perl as the
trigger, so without --all-proposed, we would fail to pick up libfoo-perl
from -proposed. Using --all-proposed for a mass-retry here again
increases the chances of the test being run correctly and getting a
passing result.

I'm sorry this is all as esoteric as it is. I've braindumped to the release
team a bit about improvements that could be made to britney to automate
retries and improve our success rate in these situations without requiring
quite so much manual intervention.


On Mon, Dec 05, 2022 at 05:04:47PM +0000, Robie Basak wrote:
> The mechansism used that translates triggers into magic "just these from
> proposed" is done via the --apt-pocket=proposed=... autopkgtest
> commandline, so the actual apt pinning is done by autopkgtest itself.
> The use of --env=ADT_TEST_TRIGGERS= etc was a red herring for me. I
> couldn't find anything in autopkgtest itself that uses this variable. I
> only saw it being used during test result collection. So what does
> ADT_TEST_TRIGGERS= do, anyway?

I didn't know the answer to this, so I googled and found
https://wiki.debian.org/debci/britneyIntegration telling me that: "This line
is the essential bit for recording the test trigger in result.tar so that
britney can match a result to a request"

And sure enough it appears in the britney source.

> However, sometimes you might find --apt-pocket=proposed=... *not* used,
> even when triggers appear to exist. I didn't get to the bottom of this.
> Maybe it's something to do with magic fallback behaviour I've heard
> mentioned (an automatic retry with different conditions?) but couldn't
> find anything documented and didn't find any implementation for this.
> For example, looking at [1], vorlon's test dated 2022-11-28 19:26:59 UTC
> and my subsequent retest dated 2022-11-30 10:20:55 UTC were subsequently
> identical, but looking at the logs, his one was called with
> --apt-pocket=proposed with no pinning defined, whereas mine was. All I
> did was hit the retry button against vorlon's test request and failure,
> and mine got run differently! I'd appreciate someone explaining this to
> me.

This seems to simply be a bug in what autopkgtest.ubuntu.com considers a
'retry', in that it is not propagating the all-proposed=1 option.

> In investigating the above, I chose to use lxd since a package
> dependency resolution issue should reproduce just fine in a container,
> and working with containers is usually much quicker. But when I couldn't
> reproduce against ubuntu-daily:lunar, I tried images:ubuntu/lunar/amd64
> for a minimalist base image, but that doesn't exist. I found this
> surprising given that images:debian/sid/amd64 does exist. I tried
> upgrading up from images:ubuntu/kinetic/amd64, and that worked (and
> didn't take long, with it being minimal). It also helped me notice the
> libssl3 thing, because openssl happens to be higher in kinetic-updates
> at the moment than the lunar release pocket, because it hasn't migrated
> from lunar proposed yet.

images: is an endpoint maintained by the LXD community and is not the base
for the images we use in autopkgtest. That's ubuntu-daily:.

Of course, at archive series opening there are bootstrapping questions about
providing those images before we've gotten the base packages updated in the
devel release pocket that would allow us to generate them.
ubuntu-daily:lunar exists now, but for reasons discussed in other threads
wrt kinetic-security vs kinetic release pocket used when bootstrapping the
autopkgtest images for the new dev series, would not necessarily be a useful
base for trying to reproduce some of these more inscrutable autopkgtest
failures.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org