Friday, 5 April 2019

Ubuntu SRUs: autopkgtest regression handling, MRE distro changes warning

Hello everyone,

Sending out this e-mail on behalf of the Ubuntu SRU team to give some
information regarding two SRU policies: clearing out some confusion
regarding autopkgtest regressions and a heads up about
distro-affecting changes in MRE-covered uploads.

It's mostly relevant only to those of you that have upload rights and
prepare SRUs on daily basis.

1. Autopkgtest Regression Handling

As most of you know (at least those that uploaded SRUs to stable
series), most SRU uploads trigger autopkgtests the same way as it
happens for uploads to the devel series. We, as the SRU team, require
the autopkgtest situation to be 'clean' before considering releasing
an update from -proposed to -updates. Whenever the SRU report page [1]
lists an ADT regression for the given upload and we have not been
provided rationale of why the regression appeared, we will usually
*not* take any action on the package. We generally require the
autopkgtest regression situation to be resolved before any action can
be performed. No upload to the stable series should introduce
regressions: doesn't matter if it's one found manually or through
automation. We'd generally like to treat manual and automated testing
the same way.

It is well known that many autopkgtest regressions reported for
selected uploads can be unrelated to the actual SRU, usually by being
already broken in the -updates pocket. This is actually a very
frequent case in the SRU world. In such cases, the uploader (or the
package validator, if having enough context) is responsible for
checking the autopkgtest regression and making sure the reported
failures are not caused by the uploaded SRU. The rationale for that
should be provided to the SRU team - best if done as a bug comment on
at least one of the bugs attached to the SRU. Once an SRU member
that's handling the update sees the rationale and is convinced by the
analysis provided, we will create a badtest hint for the failure and
proceed with releasing of the update per usual process. The uploader
can of course also make the hint change, propose a merge and include
the MP link in the bug as part of the rationale, but that is *not* a
requirement from our side.

Please remember that resolving autopkgtest failures this way requires
the uploader providing convincing argumentation and analysis of the
failure. It's not just a game of blindly hinting issues on gut feeling
only.

In case where the reported regression is a real regression, this
results in the upload becoming verification-failed and requiring a new
upload with the regression fixed. Just like with regular regressions.

I have updated the StableReleaseUpdates [2] policy document adding a
section about Autopkgtest Regressions, but we'll probably still have
to iterate on the wording of that.


2. Minor (Stable) Release Exception covered uploads and distro-affecting changes

This one is still a bit fresh and still not entirely part of the
official policy, but will become one once we get it formally worded.
It's more of a heads-up about what's coming, although we'd like all
MRE uploaders to start considering this as soon as possible. Please
forgive me if things sound a bit wavey for now.

During our last SRU team meeting we had in Malta, in light of some
recent regressions in packages that are covered by MRE's (to those
that don't know: exceptions allowing new upstream releases on special
terms), we have decided to request some additional information for
SRUs of this type.

Since new upstream release uploads can be really hard to review due to
the amount of code changing (as for new minor or even major versions
being pulled in), this creates additional risk of regressions that
could have been potentially caught during a regular SRU review.
Because of that, we would like to ask uploaders of such packages to
document any changes to the interaction of the package with the
distribution. We're still working on a formal definition of those
changes, but it's something like: any systemd unit changes visible
from the distro side, interaction with other services etc. This is so
that we, as the SRU team, can have more context on what is the
rationale for the selected changes and possibly helping out in us
making sure the new upstream version would not regress after landing
in the archive.

Such changes would be required to be stated in either the SRU tracking
bug or, in case of bigger changes, in a separate bug attached to the
upload.

As said, we still have to officially formulate the ruling, but we'd
appreciate your cooperation even now. When preparing your new upstream
version upload, think about how the new upload changes the way the
package interacts with the system and try documenting that in the
tracking bug.


So that's basically it. As a final word I'd like to thank all you
brave uploaders that fix bugs in stable series and make sure they're
up-to-date. Let's continue working together on making Ubuntu stable
releases shine!

Cheers,

[1] https://people.canonical.com/~ubuntu-archive/pending-sru.html
[2] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

--
Łukasz 'sil2100' Zemczak
Foundations Team
lukasz.zemczak@canonical.com
www.canonical.com

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel