Monday, 12 March 2018

zstd compression for packages

Hey folks,

We had a coding day in Foundations last week and Balint and Julian added support for zstd compression to dpkg [1] and apt [2].


Zstd is a compression algorithm developed by Facebook that offers far
higher decompression speeds than xz or even gzip (at roughly constant
speed and memory usage across all levels), while offering 19 compression
levels ranging from roughly comparable to gzip in size (but much faster)
to 19, which is roughly comparable to xz -6:

In our configuration, we run zstd at level 19. For bionic main amd64,
this causes a size increase of about 6%, from roughly 5.6 to 5.9 GB.
Installs speed up by about 10%, or, if eatmydata is involved, by up to
40% - user time generally by about 50%.

Our implementations for apt and dpkg support multiple frames as used by
pzstd, so packages can be compressed and decompressed in parallel

We are considering requesting a FFe for that - the features are not
invasive, and it allows us to turn it on by default in 18.10.

Balint and Julian

Raw Measurements
All measurements where performed on a cloud instance of bionic, in a basic bionic schroot with overlay, on an ssd.

Kernel install (eatmydata, perf report, time spent in compression)
Before: 54.79%
After: 11.04%

Kernel install (eatmydata)

12.49user 3.04system 0:12.57elapsed 123%CPU (0avgtext+0avgdata 68720maxresident)k
0inputs+1056712outputs (0major+159306minor)pagefaults 0swaps

5.60user 2.33system 0:07.07elapsed 112%CPU (0avgtext+0avgdata 81388maxresident)k
0inputs+1108720outputs (0major+171171minor)pagefaults 0swaps

8.80user 3.57system 0:37.17elapsed 33%CPU (0avgtext+0avgdata 25260maxresident)k
8inputs+548024outputs (0major+376614minor)pagefaults 0swaps

4.52user 3.30system 0:33.14elapsed 23%CPU (0avgtext+0avgdata 25152maxresident)k
0inputs+544560outputs (0major+386394minor)pagefaults 0swaps

firefox eatmydata
8.79user 2.87system 0:12.43elapsed 93%CPU (0avgtext+0avgdata 25416maxresident)k
0inputs+548016outputs (0major+384193minor)pagefaults 0swaps
4.24user 2.57system 0:08.54elapsed 79%CPU (0avgtext+0avgdata 25280maxresident)k
0inputs+544584outputs (0major+392117minor)pagefaults 0swaps

22.51user 7.65system 1:28.34elapsed 34%CPU (0avgtext+0avgdata 64856maxresident)k
0inputs+1376160outputs (0major+1018794minor)pagefaults 0swaps

11.34user 6.66system 1:18.04elapsed 23%CPU (0avgtext+0avgdata 64676maxresident)k
16inputs+1370112outputs (0major+1024989minor)pagefaults 0swaps

libreoffice eatmydata
22.41user 6.82system 0:27.45elapsed 106%CPU (0avgtext+0avgdata 64772maxresident)k
0inputs+1376160outputs (0major+1035581minor)pagefaults 0swaps

10.86user 5.78system 0:17.70elapsed 94%CPU (0avgtext+0avgdata 64800maxresident)k
0inputs+1370112outputs (0major+1043637minor)pagefaults 0swaps

debian developer - | - free software dev
ubuntu core developer i speak de, en

ubuntu-devel mailing list
Modify settings or unsubscribe at: