Saturday, 9 December 2017

Re: need help resolving python-setuptools backport fail

Hi Walter,

On Tue, Dec 05, 2017 at 10:29:29AM -0800, Walter Lapchynski wrote:
> On Tue, 05 Dec 2017 10:27:14 +0100 Oliver Grawert <[email protected]>
> said:
> > Am Donnerstag, den 30.11.2017, 09:25 -0800 schrieb Walter Lapchynski:
> >> I'm currently trying to backport offlineimap from zesty to trusty.
> >> This
> >> may be a bit ambitious given the python depends, but I'm taking it as
> >> a
> >> good learning experience.
> > did you consider a snap ?

> No, I did not. What guarantee would there be that a user with Trusty
> would be able to even find it? If they have trusty-backports installed
> and there's standard packaging, they'll automagically get the update.
> Not so with snaps. Snaps are nice and all, but they don't seem really
> ideal as a backport solution for this very reason. If this is a common
> response to backports, I guess it explains why there are so many
> unanswered backport requests against Trusty. Maybe if there were whole
> desktop systems completely maintained through snaps only, this might be
> a reasonable approach, but we don't have that.

Suite: trusty-backports
NotAutomatic: yes
ButAutomaticUpgrades: yes

(http://archive.ubuntu.com/ubuntu/dists/trusty-backports/Release)

You certainly would not automatically get the update to a version of the
package simply as a result of having trusty-backports installed. This is a
difference by design between SRUs and the backports repository.

In that respect, I think that snaps are at least as good a solution as
backports for this kind of thing. Backports also has the problem that
you only have two choices for the update track: you can stick with the main
Ubuntu archive and receive only security updates and SRUs, or you can switch
to backports and then get automatically updated to whatever version has been
backported most recently - even if that backport channel switches to
backporting from a newer release and causes integration problems. With
snaps, you can provide tracks for however many stable versions as might be
appropriate for the package.

It's true that the user would need to have snapd installed, which it isn't
by default on trusty, in order to know the package is available. (NB: you
also have to be using the hwe kernel to use snapd on trusty, as the GA
kernel is too old.) But -backports is also opt-in, so I wouldn't view that
as a major advantage either.

> Furthermore, that this really does not answer the original question. I
> find it quite possible that the question will still stand regardless of
> whether or not I considered a snap. This is a build-level issue, from
> what I can tell, not necessarily a matter of the packaging framework.
> That said, do you have any relevant advice?

Well, except that you don't have to backport the stack package-by-package
onto trusty in the case of a snap, you could simply use all of the
already-successfully-built .debs from zesty as needed; so I would expect
this to be a non-issue for a snap.

As for solving this in a .deb backport, I'm afraid I don't have any insight.

Cheers,
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
[email protected] [email protected]

Friday, 8 December 2017

Re: Build profile for nopcre2 build in Ubuntu

Context: https://bugs.launchpad.net/ubuntu/+source/sbuild/+bug/1734339

Hi Jonathan,

On Fri, Dec 08, 2017 at 12:47:44PM -0800, Jonathan Nieder wrote:

> In Debian, occasionally a package needs different build-time
> dependencies per target environment, due to missing packages, behavior
> differences, or other reasons. Most of the time, developers don't run
> into such issues, but when they do, they're able to resolve them using
> arch-qualified build-depends. sbuild respects arch-qualifications so
> this results in a working build on all relevant arches.

> In [1] I ran into a similar issue: Ubuntu is not able to use the git
> package from Debian because

> - in Debian, the package uses Build-Depends to allow builds against
> pcre2 and older pcre. The first alternative is pcre2, so that is
> what Debian uses (good). In backports, pcre2 is not available so
> it falls back to the older pcre (also good).

> - Ubuntu has pcre2 but it is not part of main[2].

> So Ubuntu patches the package to build against the old version of
> pcre. That alone would be fine, but it results in the package in
> Ubuntu falling out of date. I would like to update the package in
> Debian to be usable in Ubuntu as-is to prevent that happening.

> If I could use

> Build-Depends: libpcre2-dev <!ubuntu> | ...fallback...

> then I'd do that and be done.

> https://wiki.debian.org/BuildProfileSpec#Derivative_specific_profiles
> discourages this application of build profiles and says to prefer
> something distribution-agnostic like "nosystemd". Fair enough: if
> I could use

> Build-Depends: libpcre2-dev <!nopcre2> | ...fallback...

> then I'd do that and be done.

> Sensible? Any downsides I'm missing?

The downsides I see to this approach are several and significant.

- The namespace of build profiles is effectively unmanaged, aside from
https://wiki.debian.org/BuildProfileSpec#Registered_profile_names which
describes itself as a "prospective list". And also specifies that
profile names are a "distribution-specific policy". This does not lend
itself to consistent semantics between Debian and Ubuntu.

- "pcre2 is not part of main" describes Ubuntu at a single point in time
and for a particular set of Ubuntu releases. It does not represent a
fixed policy for "all releases currently built in Launchpad", and when
pcre2 does eventually supersede pcre in main, this will happen only for
later releases of Ubuntu - not for all releases that need to be buildable
with a given deployment of sbuild on the Launchpad autobuilders. This
makes encoding this information in sbuild awkward and unnatural.

- Launchpad is also used to build software that is not part of Ubuntu
proper; personal package archives (ppas) are used to build a lot of
packages without regard to what is supported in Ubuntu main. Requiring
users to care about build profiles in order to enable pcre2 for a package
in main seems like an unnecessary indirection.

The traditional solution to this problem is to provide some sort of
metapackage or virtual package to use as a build-dependency, and have this
point to the implementation of choice in each distribution. I think the
traditional solution here remains the best solution.


As an aside, your assertion in
https://bugs.launchpad.net/ubuntu/+source/git/+bug/1729075 is that having a
delta has led to the package being out of date. While it's true that the
package is currently out of date in bionic, bionic is currently in
development, and we would normally expect to see this package updated at
some point before the next stable release in April as a matter of course.
And with regards to Ubuntu 17.10, the single version that was uploaded to
Debian unstable prior to Ubuntu feature freeze, and not merged into Ubuntu
for artful, is git 1:2.14.1-2. I don't think this strongly supports the
notion that the delta is causing a drag on package currency in Ubuntu
releases.

Cheers,
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
[email protected] [email protected]

Re: Build profile for nopcre2 build in Ubuntu

Jonathan Nieder wrote:
> (please cc me on replies, since I am not subscribed)
> Hi,
>
> Context: https://bugs.ubuntu.com/1729075

This was meant to be a link to
https://bugs.launchpad.net/ubuntu/+source/sbuild/+bug/1734339
Sorry for the confusion.

> In Debian, occasionally a package needs different build-time
> dependencies per target environment, due to missing packages, behavior
> differences, or other reasons. Most of the time, developers don't run
> into such issues, but when they do, they're able to resolve them using
> arch-qualified build-depends. sbuild respects arch-qualifications so
> this results in a working build on all relevant arches.
>
> In [1] I ran into a similar issue: Ubuntu is not able to use the git
> package from Debian because
>
> - in Debian, the package uses Build-Depends to allow builds against
> pcre2 and older pcre. The first alternative is pcre2, so that is
> what Debian uses (good). In backports, pcre2 is not available so
> it falls back to the older pcre (also good).
>
> - Ubuntu has pcre2 but it is not part of main[2].
>
> So Ubuntu patches the package to build against the old version of
> pcre. That alone would be fine, but it results in the package in
> Ubuntu falling out of date. I would like to update the package in
> Debian to be usable in Ubuntu as-is to prevent that happening.
>
> If I could use
>
> Build-Depends: libpcre2-dev <!ubuntu> | ...fallback...
>
> then I'd do that and be done.
>
> https://wiki.debian.org/BuildProfileSpec#Derivative_specific_profiles
> discourages this application of build profiles and says to prefer
> something distribution-agnostic like "nosystemd". Fair enough: if
> I could use
>
> Build-Depends: libpcre2-dev <!nopcre2> | ...fallback...
>
> then I'd do that and be done.
>
> Sensible? Any downsides I'm missing?

Thanks,
Jonathan

> [1] https://bugs.launchpad.net/ubuntu/+source/git/+bug/1729075
> [2] https://bugs.launchpad.net/ubuntu/+source/pcre2/+bug/1636666

--
ubuntu-devel mailing list
[email protected]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Build profile for nopcre2 build in Ubuntu

(please cc me on replies, since I am not subscribed)
Hi,

Context: https://bugs.ubuntu.com/1729075

In Debian, occasionally a package needs different build-time
dependencies per target environment, due to missing packages, behavior
differences, or other reasons. Most of the time, developers don't run
into such issues, but when they do, they're able to resolve them using
arch-qualified build-depends. sbuild respects arch-qualifications so
this results in a working build on all relevant arches.

In [1] I ran into a similar issue: Ubuntu is not able to use the git
package from Debian because

- in Debian, the package uses Build-Depends to allow builds against
pcre2 and older pcre. The first alternative is pcre2, so that is
what Debian uses (good). In backports, pcre2 is not available so
it falls back to the older pcre (also good).

- Ubuntu has pcre2 but it is not part of main[2].

So Ubuntu patches the package to build against the old version of
pcre. That alone would be fine, but it results in the package in
Ubuntu falling out of date. I would like to update the package in
Debian to be usable in Ubuntu as-is to prevent that happening.

If I could use

Build-Depends: libpcre2-dev <!ubuntu> | ...fallback...

then I'd do that and be done.

https://wiki.debian.org/BuildProfileSpec#Derivative_specific_profiles
discourages this application of build profiles and says to prefer
something distribution-agnostic like "nosystemd". Fair enough: if
I could use

Build-Depends: libpcre2-dev <!nopcre2> | ...fallback...

then I'd do that and be done.

Sensible? Any downsides I'm missing?

Thanks,
Jonathan

[1] https://bugs.launchpad.net/ubuntu/+source/git/+bug/1729075
[2] https://bugs.ubuntu.com/1636666

--
ubuntu-devel mailing list
[email protected]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Recommending people Snap things in response to valid packaging problems (WAS: Re: need help resolving python-setuptools backport fail)

hi,
Am Freitag, den 08.12.2017, 11:50 +0100 schrieb Oliver Grawert:

> the alternative to that store would be a PPA where you put your trust
> into random people you dont know and potentially get untested changes
> to your underlying system by some dependencies they provide.

this pargraph came out a little softer than it should be actually ... 

with "trust" above i mean indeed "ultimate trust" ... 100% full root
access to all of your OS.

debhelper runs as root and so do all preinst/postinst maintainer
scripts of any deb you install.

ciao
oli

Re: Recommending people Snap things in response to valid packaging problems (WAS: Re: need help resolving python-setuptools backport fail)

hi,
Am Donnerstag, den 07.12.2017, 09:08 -0700 schrieb Matt Welland:
> On Thu, Dec 7, 2017 at 4:31 AM, Oliver Grawert <[email protected]>
> wrote:
> > 
> >
> [snip]

> > >
> > is it a bad thing to have alternatives ? 
>
> Alternatives can be a bad thing if they dilute the effort toward
> solving the problem in a robust way. If the creation of side packages
> using alternative systems subtracts from the effort put toward making
> a deb package then there is something lost from the point of view of
> Ubuntu or Debian. However that is not the only possible outcome. It
> might be that what happens is these interim solutions act as a
> stepping stone to getting good software packaged and that in turn
> leads to deb packages, an all round win.

well, the question here was about backporting an app 8 releases with
backporting a portion of python modules that will, while making
offlineimap work, potentially break other apps using these python
modules. The question was not: 

"i want to update offlineimap in bionic and need some help also
updating the python libs" 

or:

"i need to SRU offlineimap into ${LTS} to fix an important bug and need
help here to get the debian packaging right"

... it was about a task where snaps are simply better suited by
isolating the shipped python bits from the existing OS libs, by
allowing you to roll with your offlineimap port, by automagically
providing your backport to *all* supported releases and flavours of
ubuntu, by allowing your users to immediately roll back to teh former
version if a new feature after an upgrade is not wanted. a snap is
definitely the better suited alternative for the requested task here.  

My answer would have been different if the question was one of the
above, but if your question contains the words "backport" or "PPA",
nowadays "snap" is definitely the better answer over potentially
breaking users installs by pulling in untested dependencies on a system
level.

>
> I don't know which of those possible outcomes is most likely. Perhaps
> folks who have seen instances of one or the other scenario play out
> can comment?
>
> The two main obstacles to packaging that I'm aware of are dependency
> challenges and any complexity related to Debian machinery. I would
> like to package up a project of mine but I have a dependency on IUP
> which is not available as a Debian package. To solve the IUP problem
> I could attempt to debify IUP but others have tried and I'm no
> expert. For my situation the snap approach might work.

there are two other major obstacles ... 

 - testing in *all* possible scenarios... especially when replacing
   something low-level ...
 - finding a reviewer, getting your changes reviewed and approved ...

both go completely away with snaps... people that are long enough with
ubuntu might remember that we had an app store for third party apps for
several years ... that used to use debian packages. finding enough
reviewers to make sure these apps are sane and dont break enduser
systems could actually take months and ended in frustration for
everyone involved ... snaps are immediately available after upload and
due to their confinement do not require reviews at all ... 

the alternative to that store would be a PPA where you put your trust
into random people you dont know and potentially get untested changes
to your underlying system by some dependencies they provide.

>
> For the long term health of Ubuntu and Debian making it easy to
> migrate from snaps to native packages might be a good move. A
> compiler or wizard that could do most of the tedious stuff would
> help.
>
> Just $0.02 from a wannabe Debian/Ubuntu packager.
>
i personally would like to see us going in the other direction, have a
rock solid underlying deb based system (desktop or server) and use
snaps for enduser applications (from GUI and CLI apps to servers).
allowing app maintainers to independently roll on top of that system
and only maintain one package for all releases (or even multiple
versions side by side if you feel like).

while you are free to compile everything from source, snaps use deb
packages during build and as shipped dependencies by default. using a
different delivery mechanism to your enduser does not mean work on deb
packages can go away, quite the opposite, we rely on a solid and stable
archive for everything in snaps. but at the same time there is now a
proper and safe answer to "why is the latest libreoffice not in trusty
!!!111one" since the libreoffice you packages for the latest release
will just be there ... everywhere ...

we need well maintained debs, that wont change, but we should also take
advantage of new systems where they really provide advantages.

ciao
oli

Thursday, 7 December 2017

Re: Recommending people Snap things in response to valid packaging problems (WAS: Re: need help resolving python-setuptools backport fail)

-----BEGIN PGP SIGNATURE-----

iQIcBAEBCAAGBQJaKf2yAAoJEOJ/LPhFjC+kXkQP+QGtjlYnj+IDcu+B0qquCND8
R0EDfEM/6l+etIoO0zsZyOF8r4aNLNJlnRR9UT/hnKmoj2bgEn4eXls6v8P+A9ZP
pnEkzCE4tPFvFM/W8p6bqeJxTcZqSz6GOu1CcJc9nSE2XJaVJhrFxcbDRFOSUwpb
sqkG8nxxzd++qIb8ulfGSn+L7mkoLbEnrRY4TczldkK3zFaIuFnjRm7+KvRBFfVZ
nw4v5Fd4G/28v7+/vycrFJeKEM69PY5JGYWKq/vYCvWT5KofTCo9+BPxYfonEQJL
ouS81crc4sgFDB/v+jhGdJcXyn/AkIJ+tbvP18Aj8AR0zF18PxqEjeYSYXxx7W0j
RViET4W5+Jt3UgGGGAqolD5QZ4S8vUw9hfp0koUI893iLcYZYGaWXxPexGOnCVk4
c6WFU54iCMruau0rrZCJyESnkk5xvr8OBiT4NXEfeA4gzLOHVHvMlcus+bqEck1k
evE7ZWRxvzRP2yNdfx5yIQg7UvvSocnkAfXmJZHil4wEtKBEJIlGYUKHkpjUl2lU
TBteGtHj+co/HsH3I7Si4cgl2QXnn/I2fKr1BZ0Wzny+Qzzy2QDrCRvwHvXcbxw6
I6yy0XDP320/OczYw6kWpwe83fs0mgbrXz0ZWKh3/WSHFkBTGtX5+HMUDtvN8mQk
5sTKAuTMPiwouNFDrP43
=Am2T
-----END PGP SIGNATURE-----
Hello Mark and Ogra,

First of all I would like to explain myself a bit more; I was just
simply frustrated after reading the original thread because I have had
increasing interactions with people in the Ubuntu community who have
just said "Snap the thing" or similar in response to people asking about
their Ubuntu deb packaging problems, and have led them through a path
completely different from the one from which they began. I have further
responses inline, but when Mark said:

> Your characterization is simply incorrect, and I hope you'll agree after
> a cup of tea and a walk :)

While tea is not really my thing, a good 9 hours of sleep and a nice cup
of coffee does wonders. ;)

And yes, while my characterization was a bit of a hyperbole, I still
observe this sort of thing going on. (Further below I explain why in my
opinion this is not a productive thing).

On 12/07/2017 05:31 AM, Oliver Grawert wrote:
> with some experience (and knowledge of the right tools) it isnt a
> "several hours experience" really ...

To be fair here, the same can be said when building a deb package.

> i remember my first deb package taking several *days* to get right, i'd
> say several hours for a new packaging format you are not familiar with
> is a pretty good outcome :)

In my opinion if someone has already started learning Debian-based
packaging and has at least basic knowledge of the anatomy (if not more
knowledge) then learning a new format just to solve their problem is
throwing them a bit of a curveball (when the documentation in my
experience does not accurately describe how to do a lot of things quite
yet), no? (Not to say it doesn't have advantages in certain cases, but
in this particular case it might not be beneficial.)

<snip />

> I also dont see how "fixing the actual problem at hand" on a package
> system level is wrong here (vs fixing the same thing by doing a lot of
> manual work), unless your goal is to entertain yourself by doing it vs
> focusing on the end result "get that stuff to your users, so they can
> use it" ... after all the result is the same for them (a working
> offlineimap with the latest version in their preferred release)

But I think turning to the mailing list with problems that have been
encountered and asking for help (when I'm fully aware Walter has
knowledge of Snap packages' existence) proves that he would like to
actually figure out the problem. In fact, he did say:

> This may be a bit ambitious given the python depends, but I'm taking
> it as a good learning experience.

(I answered the remaining part of the email as part of my response to
Mark below)


On 12/07/2017 06:42 AM, Mark Shuttleworth wrote:
<snip />

> First, we continue to do a ton of SRUs and backporting in the deb world.

While I do certainly see the SRUs, I have seen much less backports being
processed than have actually been filed (in fact, right after my email,
I saw processing of a couple bugs pointed out to me by other developers,
which is a good step in the right direction). I don't personally think
this is a result of Snap packages existing but rather as a result of the
people in ~ubuntu-backports not having an interest in processing
backports as much anymore.

Please, do correct me if I'm wrong here.

> Second, the *actual* problem you stated is that you want to make a newer
> version of offlineimap available to Ubuntu users. And Ogra (who has been
> an Ubuntu developer as long as I have been an Ubuntu user ;))
> recommended that you try snaps, which solve that problem perfectly. They
> solve that problem for 14.04, 16.04, and current 18.04 too, which is
> very elegant.>
> The deb world is wonderful. But backporting trees of Python dependencies
> is risky for a LOT of software and a LOT of users. Sometimes it's the
> right answer, hopefully you would agree that in this case, perhaps
> digging into that snap is worth considering. Another community member
> has already made one for you to try.

To be 100% clear, this was Walter Lapchynski looking to solve his
problem, not me. ;)

In this particular case though, as I said above, his goal seems to me to
be less of "I want to get this out to lots of users quickly" and more of
"I want to learn how to fix this problem to further my packaging
knowledge" and while I feel both circumstances have their own respective
paths to follow, in this case figuring out the answer to the problem he
is having would help him reach his end goal more appropriately.

> The great thing about Ubuntu is that it consists of lots of people who
> have *different* interests, and occasionally they can teach each other
> stuff. You've helped LOTS of people learn stuff too :)

I agree, and I think that this is a major reason of why I continue to
contribute to Ubuntu; people can work on what they would like to work
on, depending on their interests and goals. I don't think Snaps are a
bad thing, I just think that they have a time and place (much like deb
packages, and other "universal" packaging formats like flatpak and
appimage, each has their advantages and disadvantages, and none of them
are complete replacements for all the rest), one that hasn't really
reached the point where I as a contributor use them enough or have
enough of a need for them to warrant contributions from my end. One
point that I would like to make is that my end goal in contributing to
this discussion and starting a new subthread (if you will) is because I
want to help Walter and anyone else that may come across this list
archive in the future solve a packaging problem and to have a discussion
about how packaging problems like this could be solved in the future.

Just my two cents, speaking for myself.

Thanks for both of your time,
--
Simon Quigley
[email protected]
tsimonq2 on freenode and OFTC
5C7A BEA2 0F86 3045 9CC8
C8B5 E27F 2CF8 458C 2FA4