We had a coding day in Foundations last week and Balint and Julian added support for zstd compression to dpkg [1] and apt [2].
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664
[2] https://salsa.debian.org/apt-team/apt/merge_requests/8
Zstd is a compression algorithm developed by Facebook that offers far
higher decompression speeds than xz or even gzip (at roughly constant
speed and memory usage across all levels), while offering 19 compression
levels ranging from roughly comparable to gzip in size (but much faster)
to 19, which is roughly comparable to xz -6:
In our configuration, we run zstd at level 19. For bionic main amd64,
this causes a size increase of about 6%, from roughly 5.6 to 5.9 GB.
Installs speed up by about 10%, or, if eatmydata is involved, by up to
40% - user time generally by about 50%.
Our implementations for apt and dpkg support multiple frames as used by
pzstd, so packages can be compressed and decompressed in parallel
eventually.
We are considering requesting a FFe for that - the features are not
invasive, and it allows us to turn it on by default in 18.10.
Thanks,
Balint and Julian
Raw Measurements
===============
All measurements where performed on a cloud instance of bionic, in a basic bionic schroot with overlay, on an ssd.
Kernel install (eatmydata, perf report, time spent in compression)
---------------------------------------------------------------------------------------
Before: 54.79% liblzma.so.5.2.2
After: 11.04% libzstd.so.1.3.3
Kernel install (eatmydata)
----------------------------------
12.49user 3.04system 0:12.57elapsed 123%CPU (0avgtext+0avgdata 68720maxresident)k
0inputs+1056712outputs (0major+159306minor)pagefaults 0swaps
5.60user 2.33system 0:07.07elapsed 112%CPU (0avgtext+0avgdata 81388maxresident)k
0inputs+1108720outputs (0major+171171minor)pagefaults 0swaps
firefox
--------
8.80user 3.57system 0:37.17elapsed 33%CPU (0avgtext+0avgdata 25260maxresident)k
8inputs+548024outputs (0major+376614minor)pagefaults 0swaps
4.52user 3.30system 0:33.14elapsed 23%CPU (0avgtext+0avgdata 25152maxresident)k
0inputs+544560outputs (0major+386394minor)pagefaults 0swaps
firefox eatmydata
-----------------------
8.79user 2.87system 0:12.43elapsed 93%CPU (0avgtext+0avgdata 25416maxresident)k
0inputs+548016outputs (0major+384193minor)pagefaults 0swaps
4.24user 2.57system 0:08.54elapsed 79%CPU (0avgtext+0avgdata 25280maxresident)k
0inputs+544584outputs (0major+392117minor)pagefaults 0swaps
libreoffice
-------------
22.51user 7.65system 1:28.34elapsed 34%CPU (0avgtext+0avgdata 64856maxresident)k
0inputs+1376160outputs (0major+1018794minor)pagefaults 0swaps
11.34user 6.66system 1:18.04elapsed 23%CPU (0avgtext+0avgdata 64676maxresident)k
16inputs+1370112outputs (0major+1024989minor)pagefaults 0swaps
libreoffice eatmydata
----------------------------
22.41user 6.82system 0:27.45elapsed 106%CPU (0avgtext+0avgdata 64772maxresident)k
0inputs+1376160outputs (0major+1035581minor)pagefaults 0swaps
10.86user 5.78system 0:17.70elapsed 94%CPU (0avgtext+0avgdata 64800maxresident)k
0inputs+1370112outputs (0major+1043637minor)pagefaults 0swaps
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
