Re: Packaging policy discussion: After=network-online.target

On Fri, May 14, 2021 at 10:14:27AM -0400, Thomas Ward wrote:
> Expanding on this Seth...

> On 5/13/21 6:51 PM, Seth Arnold wrote:
> > In the last week I've seen four different conversations about how to
> > properly start a service only 'after the network is up', and the different
> > people had different ideas of what this meant for their service:

> > - one wanted LAN up and working, nothing fancy
> > - one wanted to wait until DNS resolution was working
> > - one wanted to wait until an ospf daemon had negotiated routing
> > tables and installed a default route
> > - one waited to wait until ntp had synced (not just started, but
> > actually synced)

> I think this is the 'can of worms' I just mentioned in #ubuntu-devel on
> IRC.  Each and every one of these specific cases would need its own network
> target or SystemD target for all those cases.  We also have the case (from a
> 2017 bug that Server Team "Won't Fix"'d) that someone wants NGINX to start
> only after DNS works and the network is 'up' (and routable).  There's no
> special targets for those 'special cases' at a SystemD level.

If you look at the definition I've proposed for network-online.target,
you'll see that this target specifically encompasses "DNS works and the
network is routable". And I believe we have this working today with both
networkd and NetworkManager; to my recollection, the cases where the
implementation doesn't match the proposed definition today all relate to
cases where network-online.target waits for too *much* rather than too
*little* and causes boot delays.

> Case #2 would require the application to have some kind of start-script that
> can check DNS and not fail on DNS resolution failure.  (Or, exit in a way
> that SystemD would retry it again after a delay - exit code != 0 and not a
> sigkill, etc., with a restart delay of, say, 15-30 seconds while depending
> on network.target or network-online.target.)  NGINX fits this case, because
> if you use a DNS name in there and it doesn't resolve, it causes a bit of an
> error at startup.

$ grep Before /lib/systemd/system/systemd-resolved.service
Before=network.target nss-lookup.target shutdown.target
$

DNS resolution is always up before network.target, let alone
network-online.target.

> Case #3 requires more than LAN up, and like case #2 would need its own
> configuration / script to check that ospf is populated and such - there's
> nothing in SystemD that governs this.

Case #3, if OSPF is the only source of a default route for the system, is
covered by the proposed definition of network-online.target. (If the system
gets a default route by some other mechanism and it is subsequently replaced
by OSPF, then network-online.target is insufficient; but I think that
qualifies as a host misconfiguration.)

> Case #4 is like Case #3 and #2, except that you have to have a tie in to
> NTP.  Which, in more modern deployments, is `systemd-timesyncd` which
> handles NTP sync.  (Or Chrony, if you're like me and run an NTP server -
> chrony provides the granularity I need for the NTP server side of things).

NTP is an entirely separate thing from networking and should be handled some
other way.

> Case #2, #3, and #4 fit the standard of "This is a non-standard
> configuration that you as the sysadmin have implemented.  If you need
> special case handling for these services, that's beyond the scope of the
> 'default' packaging/service configuration goals."

For case #4, yes; and this is orthogonal to discussions of the
network-online.target.

For cases #2 and #3, I disagree per the above.

> I do agree we need a standard defined for "What does 'online' mean, and
> everyone has to accept that as the definition for what SystemD's
> 'network-online' state is.  But for now, until that standard is defined
> somewhere upstream, we have to accept that there is no standard

It doesn't have to be adopted upstream for us to be making Ubuntu better for
our users. (The goal is to get this agreed with the systemd upstream
community; but that should not be a blocker.)

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org

Re: Question to flavors: touch-base on flavor participation for 21.10!

21.10 plans and development is well underway for Ubuntu Budgie.   Looking forward to this particular journey with our growing team.

On Fri, 14 May 2021, 16:40 Erich Eickmeyer, <eeickmeyer@ubuntu.com> wrote:

On Friday, May 14, 2021 2:57:25 AM PDT Lukasz Zemczak wrote:
> Hello flavors!
>
> As we do around the start of every new cycle, I am reaching out to all
> the current official Ubuntu flavors to confirm active participation in
> the upcoming Ubuntu release - 21.10.
>
> Working towards a release requires a lot of effort, so we'd like to
> make sure all the flavors are ready, properly staffed and have enough
> time allocated to make 21.10 happen for their users. This is why,
> similarly to last year, I will need a confirmation follow-up message
> about impish participation from every flavor, that is:
>
>  * Kubuntu
>  * Xubuntu
>  * Ubuntu Studio
>  * Lubuntu
>  * Ubuntu Kylin
>  * Ubuntu MATE
>  * Ubuntu Budgie
>
> If you have any concerns regarding your participation, feel free to
> reach out to me or anyone else from the ubuntu-release team.
>
> Thank you!
>
> Cheers,

Studio is still alive and well, having recovered very well from its near-death
experience. :)

--
Erich Eickmeyer
Project Leader - Ubuntu Studio
Member - Ubuntu Community Council

+1 maintenance report

Here's my report for the (short) week of May 10-15 (Thu was a national holiday).


The week started by having a small merge party with the foundations team 🎉,
distracting me ("team +1 work") from my actual +1 work a bit, by merging:
* popularity-contests / 1.71ubuntu1, migrated
* vim / 2:8.2.2434-3ubuntu1, migrated
* curl / 7.74.0-1.2ubuntu1, pending migration for "hyphy" autopkgtest


### atlas ###

atlas / 3.10.3-10ubuntu1 ftbfs on riscv64

This FTBFS happened only on Risc-V, after some debugging I was able to reproduce
the problem locally (on amd64) when building with 'DEB_BUILD_OPTIONS=nocheck'.
I confirmed the problem is related to the build environment, by building Debian's
package (no Ubuntu modifications) in Bileto: It builds fine on Debian's buildd
on riscv but fails on Launchpad's buildd (due to building with the 'nocheck'
profile).
Trying to skip the libatlas-test package via "Build-Profiles: <!nocheck>" [0]
was a red herring, that spec does not seem to be (fully?) implemented.
https://wiki.debian.org/BuildProfileSpec
In the end I force executed the 'make check/ptcheck' targets via debian/rules
so that the relevant files for libatlas-test are always build (and tests run...)
@xnox suggested that exporting/overriding a new DEB_BUILD_OPTIONS variable at
the top of debian/rules might have worked as well, @rbalint pointed me to dpkg's
1.20.9ubuntu1 changelog where the override is described as:
DEB_BUILD_OPTIONS := $(filter-out nocheck,$(DEB_BUILD_OPTIONS))
So I implemented this improved solution as well in 3.10.3-10ubuntu3. Also, I
filed a bug at Debian, to find a proper solution to this problem:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988370


### jack-rack ###

jack-rack / 1.4.8~rc1-2ubuntu4 python2-rm transition

Removal bug filed, as this is blocking the python2-rm transition, has been
removed from Debian testing/unstable and is unmaintained upstream.
https://bugs.launchpad.net/ubuntu/+source/jack-rack/+bug/1928087


### kpatch ####

kpatch / 0.8.0-0ubuntu7 python2-rm transition

This package depends on python2-dev, upstream just changed it to python3-dev.
Changed the dependency to python3-dev as well and the package still builds fine.
No autopkgtests are provided for verification.
https://github.com/dynup/kpatch/commit/4df66fa15f95c86946050ae969658c17caebc0d2


### cdrkit ###

cdrkit / 9:1.1.11-3.2 sponsoring sync

The Debian package contains all relevant changes from Ubuntu. The delta is not
needed anymore, as explained by @waveform:
https://bugs.launchpad.net/ubuntu/+source/cdrkit/+bug/1927944


### python-pyscss ###

python-pyscss / 1.3.7-3 sponsoring sync

The Debian package contains all relevant changes from Ubuntu and is in better
shape. Sponsoring for @logan.
https://bugs.launchpad.net/ubuntu/+source/python-pyscss/+bug/1922971


### python-barbicanclient ###

python-barbicanclient / 5.0.1-0ubuntu1 ftbfs on amd64 (any)

Reviewed and sponsored a debdiff from @hellsworth.
https://bugs.launchpad.net/ubuntu/+source/python-barbicanclient/+bug/1923626


### db5.3 ###

db5.3 / db5.3 5.3.28+dfsg1-0.8 sponsoring merge

Reviewed and sponsored a debdiff from @waveform.
https://bugs.launchpad.net/ubuntu/+source/db5.3/+bug/1927978


### ply ###

ply / 3.11-4 python2-rm transition

I've dropped the python-ply package and python2 build-depends + python2
tests (in autopkgtest) for this package, to satisfy the python2-rm
transition. Also, I've sent the patch to Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=937304


### pycparser ###

pycparser / 2.20-3 python2-rm transition

I've dropped the python-pycparser package and python2 build-depends + python2
tests (in autopkgtest) for this package, to satisfy the python2-rm
transition. Also, I've sent the patch to Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=937411


### pypy & pypy3 ###

pypy+pypy3 / 7.3.4+dfsg-2 python2-rm transition

There are some issues with pypy/pypy3. According to @tumbleweed pypy is used to
build pypy3. And pypy is also used to build pypy itself (it can be bootstrapped
using python2.7, tho). Neither of both can be built using python3. Upstream
is apparently working on this Python3 support, but they are not quite ready:
https://foss.heptapod.net/pypy/pypy/-/commits/branch/rpython3/

The pypy maintainer @tumbleweed is pretty active and suggested using the
pycparser bundled with pypy, to avoid that dependency. Also, a python2.7
source could be bundled with pypy to avoid that dependency as well, if need be
(i.e. after src:python2.7 is removed).

It builds fine without the python-pycparser build-dependency (for non 'stage1'
build profiles, i.e. building with pypy with pypy itself). @tumbleweed wants to
look into fixing the '--profile=stage1' bootstrapping build using the bundled
pycparser.


### hyphy ###

hyphy / 2.5.28+dfsg-3 autopkgtest failure on amd64

The tests for this package crash with "Illegal instruction (core dumped)"
("Using /usr/lib/hyphy/bin/hyphy-avx"). I was not able to reproduce the issue
locally in qemu, so this seems to be an infrastructure issue. The test first
failed for the new "curl" upload, but I did a baseline test against "hello",
which shows that this package regressed in -release, independently of "curl".
The tests still pass on hirsute-release, tho. I'm not really sure how to
continue here...
Did something change wrt. the AVX instruction set on our amd64 test runners?


### nss (dogtag-pki) ###

nss / 2:3.63-1ubuntu1 (vs dogtag-pki) autopkgtest failure on s390x

This problem is unrelated to the s390x autopkgtest failures we had some while
ago. The 389-ds-base patch (https://github.com/389ds/389-ds-base/pull/4573) is
included in the 389-ds-base package. The dogtag-pki test passes if tested
against "389-ds-base"; it passes if tested against "hello" (using old "nss");
it fails when tested using the new "nss" (2:3.63-1ubuntu1).
Looking into DebianCI, the new "nss" passes testing with "dogtag-pki" on
s390x, tho. So this seems to be a real regression in nss 2:3.63-1ubuntu1, most
probably inside the ubuntu delta itself.
It's kind of hard to debug this right now, as Canonistack is still flaky, there
is no proper access to s390x machines... Also, I'm running out of time, so this
is something to pick up next time.


### TODO ###

- Check back with @tumbleweed about the pypy/stage1 build
- Figure out what changed wrt. AVX instruction set on the test runners
- Verify the Ubuntu delta of nss 2:3.63-1ubuntu1 regressed on s390x (dogtag-pki)


Cheers,
  Lukas

Re: Question to flavors: touch-base on flavor participation for 21.10!

On Friday, May 14, 2021 2:57:25 AM PDT Lukasz Zemczak wrote:
> Hello flavors!
>
> As we do around the start of every new cycle, I am reaching out to all
> the current official Ubuntu flavors to confirm active participation in
> the upcoming Ubuntu release - 21.10.
>
> Working towards a release requires a lot of effort, so we'd like to
> make sure all the flavors are ready, properly staffed and have enough
> time allocated to make 21.10 happen for their users. This is why,
> similarly to last year, I will need a confirmation follow-up message
> about impish participation from every flavor, that is:
>
> * Kubuntu
> * Xubuntu
> * Ubuntu Studio
> * Lubuntu
> * Ubuntu Kylin
> * Ubuntu MATE
> * Ubuntu Budgie
>
> If you have any concerns regarding your participation, feel free to
> reach out to me or anyone else from the ubuntu-release team.
>
> Thank you!
>
> Cheers,

Studio is still alive and well, having recovered very well from its near-death
experience. :)

--
Erich Eickmeyer
Project Leader - Ubuntu Studio
Member - Ubuntu Community Council

Re: Packaging policy discussion: After=network-online.target

Expanding on this Seth...

On 5/13/21 6:51 PM, Seth Arnold wrote:

  In the last week I've seen four different conversations about how to  properly start a service only 'after the network is up', and the different  people had different ideas of what this meant for their service:    - one wanted LAN up and working, nothing fancy  - one wanted to wait until DNS resolution was working  - one wanted to wait until an ospf daemon had negotiated routing    tables and installed a default route  - one waited to wait until ntp had synced (not just started, but    actually synced)

I think this is the 'can of worms' I just mentioned in #ubuntu-devel on IRC.  Each and every one of these specific cases would need its own network target or SystemD target for all those cases.  We also have the case (from a 2017 bug that Server Team "Won't Fix"'d) that someone wants NGINX to start only after DNS works and the network is 'up' (and routable).  There's no special targets for those 'special cases' at a SystemD level.

Not sure there's a way to actually handle all these cases.

We also have to be careful here: "Network Online" is, by FreeDesktop standards: "...the definition of "up" is defined by the network management software."

None of those other components mentioned (DNS, OSPF, NTP) are **network management** software.  Unless there's a standard for "online" written somewhere that isn't "your system has a routable IP assigned and the link status of the interface is UP", I don't think we can handle all the edge cases.

To be more specific, I think we need to step back from the semantics/argument regarding the target, and examine all the individual cases from the perspective of "Has the sysadmin of these systems changed the system services and configurations from the default such that it's an edge case we cannot predict or adapt to for out of the box setups?"

In the case of Case #1 in your email, that's just network-online.target as written.  But 'their service' can be overridden in SystemD to be customized to have that target.

Case #2 would require the application to have some kind of start-script that can check DNS and not fail on DNS resolution failure.  (Or, exit in a way that SystemD would retry it again after a delay - exit code != 0 and not a sigkill, etc., with a restart delay of, say, 15-30 seconds while depending on network.target or network-online.target.)  NGINX fits this case, because if you use a DNS name in there and it doesn't resolve, it causes a bit of an error at startup.

Case #3 requires more than LAN up, and like case #2 would need its own configuration / script to check that ospf is populated and such - there's nothing in SystemD that governs this.

Case #4 is like Case #3 and #2, except that you have to have a tie in to NTP.  Which, in more modern deployments, is `systemd-timesyncd` which handles NTP sync.  (Or Chrony, if you're like me and run an NTP server - chrony provides the granularity I need for the NTP server side of things).

---

Case #2, #3, and #4 fit the standard of "This is a non-standard configuration that you as the sysadmin have implemented.  If you need special case handling for these services, that's beyond the scope of the 'default' packaging/service configuration goals."  Using this argument as a basis, we could then go back to $SYSADMIN and say "We're not going to fix this in the packaging, because your use case goes beyond the goal of network-online.target as defined by FreeDesktop:  {Quote Goes Here of network-online.target data from https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/}.  Pursuant to this quote, if you want to rely on something other than the network management software's definition of 'online', you will need to change your services on your end and how they work.  This, however, is not a general packaging change that we are going to adopt for ${PACKAGE} at this time."

That, actually, is what we did with NGINX - back in 2017 it was requested to change the target, and after digging and discussion we decided we weren't going to make that change because there is no standard definition of what "online" is beyond the fact that "it's dependent on the network management software on system to identify what 'online' means".  And that was too vague a definition for us to support changing NGINX to network-online.target.  This argument continues to hold today.

I have several opinions on this, but my primary opinion is, for now, until there's a 'standard' defined at the SystemD level for what "online" should be and how that's reported back, we should reject these requests with a message like above, stating that "this is a non-standard change from the defaults, and if you need network-online.target put it in your service overrides.  if you need more than network-online.target that your network management software that configures your interfaces needs, then you will need to customize the service more yourself, and that's beyond the scope of the packaging done by Ubuntu."

I do agree we need a standard defined for "What does 'online' mean, and everyone has to accept that as the definition for what SystemD's 'network-online' state is.  But for now, until that standard is defined somewhere upstream, we have to accept that there is no standard, and we aren't going to adapt to everyone's needs and edge cases once they steer away from the 'default' configurations and network stack/configuration requirements of "the interface is up and has an IP" that seems to be the 'usual' for network management software (as stated in the SystemD documentation).

Thomas

Question to flavors: touch-base on flavor participation for 21.10!

Hello flavors!

As we do around the start of every new cycle, I am reaching out to all
the current official Ubuntu flavors to confirm active participation in
the upcoming Ubuntu release - 21.10.

Working towards a release requires a lot of effort, so we'd like to
make sure all the flavors are ready, properly staffed and have enough
time allocated to make 21.10 happen for their users. This is why,
similarly to last year, I will need a confirmation follow-up message
about impish participation from every flavor, that is:

* Kubuntu
* Xubuntu
* Ubuntu Studio
* Lubuntu
* Ubuntu Kylin
* Ubuntu MATE
* Ubuntu Budgie

If you have any concerns regarding your participation, feel free to
reach out to me or anyone else from the ubuntu-release team.

Thank you!

Cheers,

--
Łukasz 'sil2100' Zemczak
Foundations Team
lukasz.zemczak@canonical.com
www.canonical.com

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Missing critical patches of several high-risk bugs

Hi Seth,

I just found out that Ubuntu is on the CVE CNA list. 

Do you think it's possible that Ubuntu could assign the CVEs for those issues directly instead of asking Google? Once the CVE is assigned, it should also not only benefit Ubuntu but also other potentially affected kernels.

On Tue, May 11, 2021 at 6:57 PM Seth Arnold <seth.arnold@canonical.com> wrote:

On Fri, May 07, 2021 at 05:47:51PM -0700, SyzScope wrote:
> This is SyzScope, a research project that aims to reveal high-risk
> primitives from a low-risk bug.

Hello, this is pretty cool stuff. Continuing on 'executing' beyond the
point when ASAN has given up has given some pretty cool results.

I think the best way to get the most benefit out of this work is to
prioritize requesting CVEs for these issues with the Google CNA. Having
these additional details clearly visible to everybody using the CVE
infrastructure would benefit not only Ubuntu but also all our friends
in the other distributions.

There's two Google CNAs registered with the CVE project:
https://cve.mitre.org/cve/request_id.html
android-cna-team@google.com
security@google.com

I'll be honest, I don't know which CNA would be better; you may need to
discuss the project with both in order to figure out how to best handle
the work.

Thanks