Friday, 16 March 2018

Re: zstd compression for packages

Hi Julian,

Thanks for posting about this. I agree that if this is landing in dpkg+apt
upstream, it's reasonable to try to get it into the 18.04 release so that it
can be used in later releases without needing a dpkg versioned pre-depends.

If we are to evaluate using zstd as the default compression in 18.10 (or
later), I think we need to consider the total install experience, and not
just look at the dpkg unpack time.

For example:

On Mon, Mar 12, 2018 at 11:06:11AM +0100, Julian Andres Klode wrote:
> firefox
> --------
> 8.80user 3.57system 0:37.17elapsed 33%CPU (0avgtext+0avgdata 25260maxresident)k
> 8inputs+548024outputs (0major+376614minor)pagefaults 0swaps
> 4.52user 3.30system 0:33.14elapsed 23%CPU (0avgtext+0avgdata 25152maxresident)k
> 0inputs+544560outputs (0major+386394minor)pagefaults 0swaps

> firefox eatmydata
> -----------------------
> 8.79user 2.87system 0:12.43elapsed 93%CPU (0avgtext+0avgdata 25416maxresident)k
> 0inputs+548016outputs (0major+384193minor)pagefaults 0swaps
> 4.24user 2.57system 0:08.54elapsed 79%CPU (0avgtext+0avgdata 25280maxresident)k
> 0inputs+544584outputs (0major+392117minor)pagefaults 0swaps

Since you don't list binary package names for kernel or libreoffice, I'll
look at firefox, which is the obvious one. The archive version of this
package is 42MiB in size in bionic. If the zstd version is 6% larger, but
takes 4 seconds less time to unpack, this means the total install time
(download+unpack) is only improved for the end user if the download speed
from the apt source is faster than (44108204 bytes * .06 * 8bits/byte / 4.03s
~=) 5.25Mbps.

Have you established that this is a typical effective download speed for
Ubuntu users? It's certainly faster than my home connection, though I also
use a local mirror to speed up installs. It may be reasonable to expect
cloud instances to have this much throughput from their mirrors, and so it
might be the sensible choice solely on that basis; I'm just checking that
it's been measured.

In other words: if we want to make this the default, we should quantify
Daniel's remark that he would prefer a 6% faster download over a 10% faster

I think we also need to look at the spread of package size increases. If 6%
is typical, are there some packages on the high end (of both absolute package
size and relative size increase) that we should exclude from switching to
zstd? We should be transparent about our analysis here.

Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer