Monday 16 December 2019

Re: i386 autopkgtest [Re: i386 in focal: an update]

acl
alsa-lib
ann
aom
apache2
apparmor
apt
atk1.0
at-spi2-atk
at-spi2-core
attr
bash
bind9
binutils
bluez
boost1.67
brotli
bubblewrap
bzip2
cairo
capnproto
ceph
cluster-glue
clutter-1.0
cmake
colord
confuse
coreutils
corosync
cunit
cups
cython
dash
datefudge
dbus
dbus-glib
dbus-python
dconf
devscripts
dietlibc
distro-info
dlm
docbook-to-man
double-conversion
doxygen
dpkg
dxvk
ed
espeak
espeak-ng
evolution-data-server
fakeroot
faketime
faudio
fenix
fenix-plugins
ffmpeg
fftw3
fig2dev
firebird3.0
flite
fluidsynth
fpc
freeglut
fribidi
game-data-packager
gcc-8
gcc-9
gcc-snapshot
gconf
gcr
gdbm
gdisk
gdk-pixbuf
gem2deb
geocode-glib
gjs
glib2.0
glibc
glib-networking
gnupg2
gnuplot
gnutls28
gobject-introspection
gpgme1.0
graphene
graphite2
graphviz
grep
gspell
gtk+2.0
gtk+3.0
gtkmm2.4
gvfs
gzip
harfbuzz
hwloc
ibus
icu
imagemagick
inetutils
initramfs-tools
init-system-helpers
iproute2
iptables
jdupes
jemalloc
json-glib
kconfig
keyutils
krb5
kronosnet
ladspa-sdk
lapack
lbfgsb
leveldb
libanyevent-perl
libarchive
libassuan
libasync-interrupt-perl
libb-hooks-op-check-perl
libbsd-resource-perl
libcaca
libcap2
libclass-xsaccessor-perl
libclone-perl
libcommon-sense-perl
libdaemon
libdatrie
libdevel-callchecker-perl
libdigest-sha-perl
libevent-perl
libev-perl
libfile-libmagic-perl
libfilter-perl
libgd2
libgdata
libgd-perl
libglib-perl
libgpg-error
libgphoto2
libhtml-parser-perl
libical3
libinput
libio-pty-perl
libiptcdata
libjsoncpp
libkate
liblinux-epoll-perl
liblist-moreutils-perl
liblocale-gettext-perl
libmicrohttpd
libnetaddr-ip-perl
libnet-ssleay-perl
libnfs
libnotify
libofa
liboggz
libomxil-bellagio
libopenmpt
libparams-classify-perl
libparams-util-perl
libparams-validate-perl
libperlio-gzip-perl
libpipeline
libproxy
libpsl
libqb
libraw
librsvg
libsass
libsdl2
libsdl-sge
libseccomp
libsecret
libsocket6-perl
libsort-key-perl
libsoup2.4
libsoxr
libssh
libstb
libsub-identify-perl
libsub-name-perl
libtaint-runtime-perl
libtest-leaktrace-perl
libtest-taint-perl
libtext-charwidth-perl
libtext-iconv-perl
libthai
libtheora
libtommath
libunicode-utf8-perl
libusb-1.0
libutempter
libuuid-perl
libuv1
libvariable-magic-perl
libvisual
libvncserver
libvorbis
libvpx
libwacom
libwant-perl
libwmf
libxcb
libxml2
libxml-libxml-perl
libxml-parser-perl
libyaml-libyaml-perl
libzstd
linux
livecd-rootfs
llvm-toolchain-9
matplotlib
mawk
mir
multipath-tools
munge
mysql-8.0
ncurses
ndctl
network-manager
nghttp2
ninja-build
nss-mdns
nss-wrapper
numpy
openjdk-lts
openldap
openssh
openssl
pacemaker
pango1.0
perl
php7.3
pillow
pinentry
policykit-1
poppler
postgresql-12
protobuf
pulseaudio
pupnp-1.8
pygobject
pypy
pyqt5
python2.7
python3.7
python3.8
python-apt
python-bcrypt
python-cffi
python-coverage
python-evdev
python-numpy
python-scandir
python-scipy
pyyaml
raqm
re2
re2c
rhash
rsync
ruby2.5
ruby-defaults
ruby-json
ruby-nokogiri
samba
sane-backends
sdlgfx
sed
snapd-glib
sonic
spdlog
stunnel4
svox
swig
systemd
systemtap
talloc
tdb
ubuntu-drivers-common
uchardet
udisks2
umockdev
update-notifier
upower
util-linux
valgrind
vim
vlc
vorbis-tools
x264
x265
xapian-core
xdg-dbus-proxy
xterm
zsh
zvbi
A further update - after some judicious pruning of the build-dependencies of
gst-plugins-bad1.0, the total number of i386 packages for autopkgtesting is
reduced from 311 to 299 (attached).

Here is the new list of packages with autopkgtests that warrant
investigation. We now have 29% of the i386 autopkgtests passing, 24%
confirmed not testable, and 47% still needing investigation.

I have also now added 'badtest' hints for the current version of each of
these remaining packages, so as of now they should no longer be causing
proposed-migration delays for any other packages. As you upload new
versions of these packages, please make sure to do the analysis of the
autopkgtest failures and work with the release team to either fix the tests
or have them permanently blacklisted, as appropriate.

success: 86
blacklisted: 73
failed, needs investigation: 140
acl
alsa-lib
ann
bind9
binutils
boost1.67
brotli
bubblewrap
bzip2
capnproto
cluster-glue
cmake
colord
confuse
cunit
cups
cython
datefudge
dbus
devscripts
dietlibc
dlm
docbook-to-man
double-conversion
doxygen
fakeroot
faketime
fenix
fenix-plugins
fftw3
freeglut
fribidi
game-data-packager
gcc-8
gcc-9
gcc-snapshot
gdbm
gem2deb
glibc
gnupg2
gpgme1.0
graphite2
grep
gtk+3.0
gvfs
gzip
ibus
icu
inetutils
init-system-helpers
iproute2
jemalloc
krb5
kronosnet
lapack
lbfgsb
libarchive
libassuan
libcaca
libcap2
libdaemon
libdatrie
libgd2
libgpg-error
libgphoto2
libiptcdata
libjsoncpp
libkate
libmicrohttpd
libnfs
libofa
libomxil-bellagio
libpipeline
libpsl
libqb
libraw
libsass
libsdl2
libsdl-sge
libseccomp
libsoxr
libthai
libtommath
libusb-1.0
libutempter
libuv1
libvisual
libvncserver
libxcb
libxml2
livecd-rootfs
llvm-toolchain-9
matplotlib
mir
multipath-tools
ncurses
network-manager
nghttp2
nss-wrapper
numpy
openjdk-lts
openssh
openssl
pacemaker
perl
pillow
pinentry
postgresql-12
protobuf
pupnp-1.8
python2.7
python3.7
python3.8
python-apt
python-bcrypt
python-cffi
python-coverage
python-evdev
python-numpy
python-scipy
pyyaml
re2
ruby2.5
ruby-json
ruby-nokogiri
samba
sane-backends
sdlgfx
stunnel4
systemd
systemtap
talloc
tdb
ubuntu-drivers-common
uchardet
udisks2
valgrind
vim
xterm
zsh

On Tue, Dec 10, 2019 at 11:03:30AM -0800, Steve Langasek wrote:
> Hi all,
>
> After a few days of bugfixing tests to be compatible with the new
> autopkgtest cross-arch infrastructure, I thought it would be useful to give
> folks some stats, and point you to a list of packages requiring further
> investigation that you can help drive down.
>
> Of 311 autopkgtests that we care about on i386, 24% are currently passing,
> 8% have been confirmed as not-cross-testable and blacklisted by the release
> team, and 68% still need investigation. Details, including a list of
> packages to investigate, produced by the attached script:

>
> success: 74
> blacklisted: 26
> failed, needs investigation: 211
> acl
> alglib
> alsa-lib
> ann
> bind9
> binutils
> boost1.67
> brotli
> bubblewrap
> bzip2
> capnproto
> cfitsio
> cluster-glue
> cmake
> colord
> confuse
> coreutils
> cunit
> cups
> cython
> datefudge
> dbus
> devscripts
> dietlibc
> distro-info
> dlm
> docbook-to-man
> double-conversion
> doxygen
> faketime
> fenix
> fenix-plugins
> fftw3
> fpc
> freeglut
> fribidi
> fyba
> game-data-packager
> gcc-8
> gcc-9
> gcc-snapshot
> gdbm
> gem2deb
> gl2ps
> glibc
> gnupg2
> gpgme1.0
> graphite2
> grep
> gtk+2.0
> gtk+3.0
> gvfs
> gzip
> harfbuzz
> ibus
> icu
> inetutils
> init-system-helpers
> iproute2
> jemalloc
> json-glib
> krb5
> kronosnet
> lapack
> lbfgsb
> libanyevent-perl
> libarchive
> libassuan
> libasync-interrupt-perl
> libb-hooks-op-check-perl
> libbsd-resource-perl
> libcaca
> libcap2
> libclone-perl
> libcommon-sense-perl
> libdaemon
> libdatrie
> libdevel-callchecker-perl
> libevent-perl
> libev-perl
> libfile-libmagic-perl
> libfilter-perl
> libgd2
> libgd-perl
> libgpg-error
> libgphoto2
> libhtml-parser-perl
> libinput
> libio-pty-perl
> libiptcdata
> libjsoncpp
> libkate
> liblist-moreutils-perl
> liblocale-gettext-perl
> libmicrohttpd
> libnetaddr-ip-perl
> libnet-ssleay-perl
> libnfs
> libofa
> libomxil-bellagio
> libopenmpt
> libparams-classify-perl
> libparams-util-perl
> libparams-validate-perl
> libperlio-gzip-perl
> libpipeline
> libproxy
> libpsl
> libqb
> libraw
> libsass
> libsdl2
> libsdl-sge
> libseccomp
> libsecret
> libsocket6-perl
> libsort-key-perl
> libsoup2.4
> libsoxr
> libssh
> libsub-identify-perl
> libsub-name-perl
> libtaint-runtime-perl
> libtest-leaktrace-perl
> libtest-taint-perl
> libtext-charwidth-perl
> libtext-iconv-perl
> libthai
> libtommath
> libunicode-utf8-perl
> libusb-1.0
> libutempter
> libuuid-perl
> libuv1
> libvariable-magic-perl
> libvisual
> libvncserver
> libvorbis
> libwacom
> libwant-perl
> libxcb
> libxml2
> libxml-libxml-perl
> libxml-parser-perl
> libyaml-libyaml-perl
> linux
> livecd-rootfs
> llvm-toolchain-9
> matplotlib
> mir
> mpi4py
> multipath-tools
> ncurses
> network-manager
> nghttp2
> nss-wrapper
> numpy
> openjdk-lts
> openssh
> openssl
> pacemaker
> perl
> pillow
> pinentry
> poppler
> postgresql-12
> protobuf
> pulseaudio
> pupnp-1.8
> python2.7
> python3.7
> python3.8
> python-apt
> python-bcrypt
> python-cffi
> python-coverage
> python-cryptography
> python-evdev
> python-lz4
> python-nacl
> python-numpy
> python-scipy
> py-ubjson
> pyyaml
> raqm
> re2
> ruby2.5
> ruby-json
> ruby-nokogiri
> samba
> sane-backends
> sdlgfx
> sed
> stunnel4
> superlu
> systemd
> systemtap
> talloc
> tdb
> twisted
> ubuntu-drivers-common
> uchardet
> udisks2
> upower
> util-linux
> valgrind
> vtk6
> vtk7
> xterm
> zope.interface
> zsh
>
> Cheers,
> --
> Steve Langasek Give me a lever long enough and a Free OS
> Debian Developer to set it on, and I can move the world.
> Ubuntu Developer https://www.debian.org/
> slangasek@ubuntu.com vorlon@debian.org
>
> On Fri, Dec 06, 2019 at 03:57:21PM -0800, Steve Langasek wrote:
> > Hi folks,
> >
> > The changes to i386 have proceeded as described in my earlier email. The
> > autopkgtest infrastructure has been converted to run tests of i386 binaries
> > on an amd64 VM, and i386 binaries not in the whitelist are now in the
> > process of being removed. As of this moment, there are 1,679 source
> > packages in the whitelist for i386, and 8,921 source packages with published
> > i386 binary packages in focal, vs. 14,344 source packages with published
> > amd64 binaries. At the current rate, I expect the binary removals to
> > complete some time tomorrow.
> >
> > On the autopkgtest side, there are now a total of 311 source packages
> > producing i386 binaries which also have autopkgtests[1]. Any packages
> > reporting autopkgtest regressions on i386 which are not on this list should
> > have their test failures ignored; if you see i386 autopkgtest failures for
> > other packages blocking migrations out of focal-proposed, please ask a
> > member of the release team to override.
> >
> > It's good that the number of remaining autopkgtests for i386 is as small as
> > it is, because while the /infrastructure/ is now capable of cross-testing
> > i386 packages on amd64, the vast majority of these tests don't run correctly
> > in such an environment without modification. I have triggered runs of all
> > of these tests on the amd64 testbed, so current status of the tests is
> > visible on autopkgtest.ubuntu.com for each package.
> >
> > Since these tests have also de facto regressed in the release pocket due to
> > changes to the infrastructure, it is appropriate for those test failures to
> > be ignored for proposed-migration purposes as well. I intend to go through
> > and systematically mark these as bad tests, but in the meantime if you have
> > package migrations that are being blocked by one of these, please reach out.
> >
> >
> > The other thing that you can do is help fix the failing autopkgtests, so
> > that we have better test coverage on i386. I've identified a few common
> > patterns so far in the test failures:
> >
> > - Tests of Essential or Priority: required/import packages that require removing
> > the amd64 package in favor of the i386 package (e.g.: bash, apt,
> > systemd). These tests will never work, and - outside of launchpad
> > builds, where we are still building i386 natively instead of on an amd64
> > host - these packages aren't relevant to the i386 experience on focal.
> > These test failures should be permanently ignored.
> >
> > - Tests with test dependencies on python on perl modules. It's a
> > longstanding known issue with multiarch that python and perl modules pose
> > problems for cross-installation, because they depend on the interpreter
> > of the matching architecture. Neither perl nor python can be
> > cross-installed on the testbed (the amd64 versions are both part of the
> > test infrastructure), so these tests will always fail with 'badpkg'.
> > While some of these tests may be more relevant to the behavior of our
> > i386 runtime libraries, fixing them is largely intractable so these are a
> > low priority.
> >
> > - Tests with other uninstallable test dependencies ("badpkg"). Many of
> > these are going to be fixable, but require either annotations of
> > additional packages in the archive as Multi-Arch: foreign (if
> > appropriate), or annotations of the test dependencies in
> > debian/tests/control with architecture information. The autopkgtest
> > dependency resolver for cross-testing uses apt-get build-dep, so that the
> > full range of multiarch build-dependency specifiers are available as
> > described at <https://wiki.ubuntu.com/MultiarchCross>.
> >
> > - Build tests. These are common tests for libraries, and some of the
> > easiest to fix, but most don't work out of the box because they don't
> > invoke the cross-toolchain. Here are some example patches for various
> > types of build systems used in build tests:
> > - pkg-config: https://bugs.debian.org/946303
> > - meson: https://bugs.debian.org/946292
> >
> > Happy fixing,
> > --
> > Steve Langasek Give me a lever long enough and a Free OS
> > Debian Developer to set it on, and I can move the world.
> > Ubuntu Developer https://www.debian.org/
> > slangasek@ubuntu.com vorlon@debian.org
> >
> > [1] See attachment. List calculated by:
> > $ join -j1 <(wget -O - -q -o /dev/null \
> > https://people.canonical.com/~ubuntu-archive/germinate-output/i386.focal/i386+build-depends.sources \
> > | awk '{ print $1 }' | sort) \
> > <(cat /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_*Sources \
> > | grep-dctrl -r -FTestsuite . -a '!' -FArchitecture -X all \
> > -sPackage -n | sort)
> >
> > On Thu, Nov 28, 2019 at 12:46:34PM -0800, Steve Langasek wrote:
> > > Thanks to thorough feedback from our community, we now have a reasonably
> > > comprehensive answer to the question of what 32-bit compatibility library
> > > packages are needed on x86 for Ubuntu 20.04.
> > >
> > > https://discourse.ubuntu.com/t/community-process-for-32-bit-compatibility/12598/46
> > >
> > > Some developers will have noticed changes this week to the behavior of focal
> > > builds in Launchpad. Out of 30,000 source packages in focal, there is now a
> > > whitelist of about 1,700 source packages which will trigger builds on i386
> > > in Launchpad. This means that other packages which previously built on i386
> > > will need to have the binaries from the old version of the package removed
> > > before they will be migratable from focal-proposed to focal.
> > >
> > >
> > > As a side note, the implementation of this also affects PPA builds, because
> > > the whitelist applies to the focal series as a whole. In general, you
> > > should not expect to need i386 builds of third-party packages in PPAs for
> > > focal either, given that i386 in focal exists solely for compatibility with
> > > legacy binary software. However, if you have a third-party package that you
> > > believe it's important to continue producing i386 binary builds of in
> > > Launchpad for Ubuntu 20.04, please contact the Ubuntu archive admins
> > > (ubuntu-release@lists.ubuntu.com, or #ubuntu-devel on freenode.net for best
> > > results), and we can evaluate including your PPA package in the whitelist.
> > >
> > >
> > > At the moment, I am doing some manual removals of the i386 binaries as I see
> > > them show up as blockers on
> > > https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html
> > > and as I'm able to determine that the removals aren't going to cause
> > > near-term knock-on problems. But if some i386 binaries aren't being removed
> > > fast enough and this is blocking your work, feel free to reach out to an
> > > archive admin to ask for their removal.
> > >
> > > In the slightly less near term, the plan is to do a mass binary removal of
> > > all of the i386 binary packages in focal built from sources other than those
> > > in the whitelist. However, before pulling the trigger on this mass removal,
> > > there are some changes that should be landed to our autopkgtest
> > > infrastructure, so that we can continue to run autopkgtests for those
> > > remaining 1700 packages. In summary: the plan is not to retain the test
> > > dependencies of those 1700 packages on i386, but instead to cross-test the
> > > i386 libraries on an amd64 host, which ultimately means testing them in an
> > > environment that better models the expected real-world usage. The work is
> > > in progress for this change and I'm currently anticipating landing it next
> > > week.
> > >
> > > In the meantime, if you need any help getting packages migrating to the
> > > focal release from -proposed, please reach out on #ubuntu-release on IRC.

Thanks,
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org