Monday 10 December 2018

autopkgtest verision deployed on production infrastructure updated

Hi all,

I've just pushed a bunch of commits to our deployed branch of
autopkgtest¹ that bring us up to date with upstream, where the
developers have done a lot of work. Many thanks to them. The most
interesting thing to subscribers to this list will be the changes -
mainly additions - to the list of known test restrictions. I've pasted
the diff of README.package-tests.rst below.

I don't currently expect any breakage to result from this, but if you
notice anything weird please reply here or come to #ubuntu-release and
we'll take a look.

Cheers,

--
Iain Lane [ iain@orangesquash.org.uk ]
Debian Developer [ laney@debian.org ]
Ubuntu Developer [ laney@ubuntu.com ]

[1] https://code.launchpad.net/~ubuntu-release/autopkgtest/+git/development/

@@ -237,14 +245,84 @@ needs-reboot
The test wants to reboot the machine using
``/tmp/autopkgtest-reboot`` (see below).

-needs-recommends
+needs-recommends (deprecated)
Enable installation of recommended packages in apt for the test
dependencies. This does not affect build dependencies.

+ Don't use this for new tests, we want to remove it altogether.
+ See https://lists.debian.org/debian-ci/2018/06/msg00016.html
+ and bug 896698 for the rationale.
+
+flaky
+ The test is expected to fail intermittently, and is not suitable for
+ gating continuous integration. This indicates a bug in either the
+ package under test, a dependency or the test itself, but such bugs
+ can be difficult to fix, and it is often difficult to know when the
+ bug has been fixed without running the test for a while. If a
+ ``flaky`` test succeeds, it will be treated like any other
+ successful test, but if it fails it will be treated as though it
+ had been skipped.
+
+skippable
+ The test might need to be skipped for reasons that cannot be
+ described by an existing restriction such as isolation-machine or
+ breaks-testbed, but must instead be detected at runtime. If the
+ test exits with status 77 (a convention borrowed from Automake), it
+ will be treated as though it had been skipped. If it exits with any
+ other status, its success or failure will be derived from the exit
+ status and stderr as usual. Test authors must be careful to ensure
+ that ``skippable`` tests never exit with status 77 for reasons that
+ should be treated as a failure.
+
+skip-not-installable
+ This test might have test dependencies that can't be fulfilled on all
+ architectures. Therefore, when apt-get installs the dependencies, it will
+ fail. Don't treat this as a test failure, but instead treat it as if the
+ test was skipped.
+
+hint-testsuite-triggers
+ This test exists purely as a hint to suggest when rerunning the
+ tests is likely to be useful. Specifically, it exists to
+ influence the way dpkg-source generates the Testsuite-Triggers
+ .dsc header from test metadata: the Depends for this test are
+ to be added to Testsuite-Triggers. (Just as they are for any other
+ test.)
+
+ The test with the hint-testsuite-triggers restriction should not
+ actually be run.
+
+ The packages listed as Depends for this test are usually indirect
+ dependencies, updates to which are considered to pose a risk of
+ regressions in other tests defined in this package.
+
+ There is currently no way to specify this hint on a per-test
+ basis; but in any case the debian.org machinery is not able to
+ think about triggering individual tests.
+
+superficial
+ The test does not provide significant test coverage, so if it
+ passes, that does not necessarily mean that the package under test
+ is actually functional. If a ``superficial`` test fails, it will be
+ treated like any other failing test, but if it succeeds, this is
+ only a weak indication of success. Continuous integration systems
+ should treat a package where all non-superficial tests are skipped as
+ equivalent to a package where all tests are skipped.
+
+ For example, a C library might have a superficial test that simply
+ compiles, links and executes a "hello world" program against the
+ library under test but does not attempt to make use of the library's
+ functionality, while a Python or Perl library might have a
+ superficial test that runs ``import foo`` or ``require Foo;`` but
+ does not attempt to use the library beyond that.
+
Defined features
----------------

-There are no currently defined Features.
+test-name
+ Set an explicit test name for the log heading and the ``summary`` file
+ for a ``Test-Command:`` inline test. When not given, these are
+ enumerated like ``command1``.