Friday 3 February 2023

PSA: autopkgtest environment behavior change

I recently discovered that autopkgtests which are getting OOM killed
were behaving differently. Paride and I have tracked this down to a
change in systemd's behavior[1] which results in the testbed exiting
abnormally. I've temporarily[2] modified[3] our autopkgtest code in
production so that this will not be considered a "testbed" failure which
will prevent packages with autopkgtests that are OOM killed from running
repeatedly.

I mention it because the log files for tests in this situation have
changed and have become less informative. Looking the log file[4] for
r-cran-rstanarm you'll see the following:

autopkgtest [18:46:11]: test run-unit-test: -----------------------]
autopkgtest [18:46:12]: test run-unit-test: - - - - - - - - - - results
- - - - - - - - - -
run-unit-test FAIL non-zero exit status 255

Which is less helpful than the previous "Killed signal terminated"
message. So when looking at proposed migration please keep this in mind.

Additionally, since we are talking about tests running out of memory I
wanted to mention that with the qemu backend it is possible to adjust
the amount of memory your virtual machine has using the "--ram-size"
argument. For example if I want to see if having more memory will allow
a test to pass I'll use the following before adding it to big_packages:

autopkgtest --apt-upgrade r-cran-rstanarm --shell-fail -- qemu
--ram-size=8192 --cpus=4 /srv/vms/autopkgtest-lunar-amd64.img

Finally, one benefit of this is that it became somewhat easier to find a
bunch of packages that would pass if given more memory. I've
investigated several of them and added them to 'big_packages'. If the
developers on +1 maintenance next week could look at the following list
it would be helpful!

amd64-igraph
amd64-juce
amd64-libgd2
amd64-libthrust
amd64-mariadb
amd64-mariadb-10.6
amd64-nvidia-cuda-samples
amd64-openjdk-8

[1]
https://github.com/systemd/systemd-stable/commit/0db0562852aafd89c84bf12cdf3de770898048d1
[2] I've spoken with the Foundations team and they plan on uploading a
new systemd.
[3]
https://git.launchpad.net/~ubuntu-release/autopkgtest/+git/development/commit/?id=842ca0f4bd78a6b0a7c1b484f7da242581651804
[4]
https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/amd64/r/r-cran-rstanarm/20230203_185207_c0b1c@/log.gz

Have a great weekend!
--
Brian Murray

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel