Thursday, 11 February 2016

Re: Archive Reorg Episode VII: Follow Build-Depends

Hi Matthias,

On Thu, Feb 11, 2016 at 12:36:27PM +0100, Matthias Klose wrote:
> On 10.02.2016 21:32, Dimitri John Ledkov wrote:
> >This does not abolish the MIR process. If a hard binary dependency is
> >gained (e.g. shared linking), the binary and source package still must
> >go through MIR process and be published in main.

> So an existing app package gains a new (universe) dependency on libfoo-dev.
> Builds fine, maybe migrates, and then image builds fail because of the
> libfoo1 component mismatch. Now you can either pre-promote the libfoo, or
> re-upload app without the dependency (if that works).

There is a third option here, which I discussed with Colin and Adam and that
(I think) we agree is workable: enable inclusion of universe in the image
builds during development, turning this off only once we are in the final
freeze for release.

This has the important properties of keeping velocity high for image daily
builds (no build failures due to component-mismatches in the devel series,
and no human intervention required to promote a package to un-break a
build), while ensuring that we are still able to track the list of
outstanding issues for release in a well-visible report.

> This probably will lead to more pre-promotions, and looking at the current
> back-log of security related MIRs the time between build and promotion
> will increase, making it probably harder to revert such a change.

In fact, as far as pre-promotions are concerned, if we enable universe for
pre-release image builds the pressure to do pre-promotions seems much lower
than it is today.

> I'm a bit worried that we'll then have to chase people to subscribe teams to
> the new packages, write the MIR, ... We'll save some time by not processing
> B-D only MIRs, but I think for the remaining MIRs we'll have to spend more
> time.

Well, first of all, the problem you're describing doesn't sound to me like
some future dystopia... it sounds to me like the current complaints about
the MIR process, with the owning teams not being reliable at taking
responsibility for their new packages and leaving the MIR team to drive

Secondly, if we eliminate the problem of image build failures driving
pre-promotions, I believe it actually becomes *easier* for teams to track
their outstanding MIRs, even without considering that the volume of packages
needing MIRs will go down. This is because c-m won't cause packages to be
stuck in -proposed, therefore we won't need to track two separate c-m
( vs.
and can instead get a single view of the problem again.

Thirdly, it should be noted that a 35% reduction of source packages in main
(Dimitri's current best estimate) does not translate to a 35% reduction in
MIRs. My guess is that the reduction in MIRs will be *greater* than 35%
compared to the current volume; you not only don't have to do MIRs any more
for the build-dependencies of the packages left in main, you also don't have
to do MIRs for the build-dependencies *or* build-dependencies of those
packages that are leaving main. So maybe the actual savings looks more like
1-(1-35%)², or 58%.

And fourthly :-), please note that at a technical and process level, this
latest proposal is almost entirely equivalent to the previous proposal for
archive reorganization. That proposal foundered mainly because of the
difficulties about communicating to our users that "main" no longer means
"supported by Canonical". But otherwise, we're talking about all the same
kinds of changes - packages in main and universe can build-depend on each
other; only packages that are going to be used at runtime need to go through
a review process, not all build-dependencies. Yes, there may be some
changes that should be made to the MIR process to properly deal with this
change in the boundaries. But the fundamental benefit here is that we're
significantly reducing the set of packages that we have to have MIR process
around, and that the security team has to provide support for post-release.
I hope you'll agree that this makes it worth pursuing, even if there wind up
being some bumps along the road for the MIR process.

> We unfortunately already have some kind of "dput and forget" attitude with
> packages staying in -proposed. This change maybe will foster an
> "pre-approve and forget" attitude.

To be frank, I consider the fact that our developers *can't* "dput and
forget" without causing problems for the archive to be a major problem with
our current tooling. Developers should not have to poll 4-5 different
webpages after every upload to find out if that package will actually make
it into the development release. We do a good job of notifying developers
of build failures (via Launchpad), but we don't notify them about:

- dep-waits
- autopkgtest failures
- proposed-migration stalls

At least for the first two of these, I think uploaders *should* be actively
notified, instead of us being surprised and outraged that developers don't
make memos to themselves to check back repeatedly for days at a time after
each upload

Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer