On Tue, Jul 11, 2023 at 12:53:00AM +0000, Seth Arnold wrote:
> On Mon, Jul 10, 2023 at 10:55:06AM +0200, Adrien Nader wrote:
> > There is a little-know but very interesting property of LZMA: its
> > decompression speed does not depend on the uncompressed size but only on
> > the compressed size. What this means is that if you compress a 100MB
> > down to 20MB, it will take roughly twice as long to decompress than if
> > you compress it down to 10MB. In other words, higher compression means
> > faster decompression.
>
> This makes a certain amount of sense -- so much of a computer's
> operational time is spent waiting for data to arrive from memory into
> the processor, refilling cache lines, etc.
>
> You nerd-sniped me into testing a bunch of algorithms on the
> firefox_115.0+build2.orig.tar from our archive.
>
> I only ran these things once, and quite a lot of them ran while a
> second one was running, but this system (dual xeon E5-2630v3) has enough
> processors and memory that it probably didn't matter much.
>
> Times in seconds, with lower level on the left, higher on the right:
>
> 1 3 5 9
> compression:
> gzip 39 46 73 211
> zstd 8 12 23 54
> bzip2 228 237 249 265
> lzma 154 294 643 945
> xz 159 298 644 945
>
> decompression:
> gzip 16 15 15 15
> zstd 3 3 3 3
> bzip2 68 73 74 75
> lzma 41 37 35 33
> xz 36 32 31 30
>
> xz of course absolutely dominates the end file sizes:
>
> 2989486080 original
>
> 515273416 xz -9
> 625958113 zstd -9
> 647365812 xz -1
> 666820870 zstd -5 (seemed like a sweet spot in the timings)
>
> Anyway it's fun to see that gzip and zstd have consistent decompression
> speeds, lzma and xz get faster as they go smaller, and bzip2 just gets
> slower the more it has to think.
One caveat to your whole experiment is that zstd -9 is roughly
comparable to gzip -9, whereas zstd -19 is the level comparable
to xz -9 (arguably you may also pick 17, 18 depending on how
long the compression takes vs xz -9 :D).
I think that's a reason you don't see speedups because zstd is already
quite fast and you are not really using high compression levels.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
