Friday, 7 July 2023

Re: Reducing initramfs size and speed up the generation



On Sat, 8 Jul 2023, 02:49 Benjamin Drung, <bdrung@ubuntu.com> wrote:
On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote:
> On Sat, 8 Jul 2023 at 01:19, Benjamin Drung <bdrung@ubuntu.com> wrote:
> >
> > Hi all,
> >
> > a year ago we changed the default compression and level for the
> > initramfs to zstd -1. This fixed the very slow creation times on
> > development boards (see bug #1958148), but that leads to bigger
> > initramfs sizes that triggered other bugs (like bug #1842320).
> > Big initramfs sizes can also fill up small sized /boot partitions easily
> > (grooming the 850 initramfs-tools bugs revealed several such reports).
> >
> > Using xz -9 would give very good compression, but it takes very long
> > (especially on slow development boards) and a lot of memory (good luck
> > on Raspberry Pis with small memory like Pi Zeros).
> >
> > I propose following approach to address the drawback: Create cpio
> > archives (compressed with xz -9) for the kernel modules and firmware
> > files when building the kernel/firmware Debian package. Then ship those
> > cpio archives in the package (or in a separate binary package). Then the
> > CPU load it put on the builders. The cpio archives would contain the
> > modules for MODULES=most.
> >
> > mkinitramfs will then look for those cpio archives and uses those in
> > case they are present. Such a initramfs would look like this:
> >
> > * AMD/Intel microcode cpio archive (on amd64)
> > * main cpio archive compressed with zstd -1
> > * kernel modules from the Debian package compressed with xz -9
> > * firmware files from the Debian package compressed with xz -9
> >
>
> Majority of our instances boot without initrd, and there too they
> don't load most of the modules.
> Creating xz -9 compressed archive of all modules, still pays the
> penalty to decompress most of them, and then not modprobe them.
> I was hoping to achieve a similar in spirit approach, but didn't quite
> have the time to implement is:
>
> 1) change linux-modules and linux-firmware to ship .ko.zst
> firmware.bin.zst compressed with zstd -19 at .deb build time
> 2) this saves install size of the packages, with only slightly
> increased download size
> 3) modify initramfs-tools to include compressed files into a separate
> initrd, which is not compressed (i.e. exclude .zst files from the
> default main compressed cpio archive, and append them in the second
> main cpio archive that is uncompressed)
> 4) this should achieve quick initrd creation, which will be smaller in
> size that current status, and will boot faster as it will only
> decompress modules/firmware it actually needs at boot
>
> For experimentation locally, you can recompress .ko with zstd in place
> in /lib/modules/; and rerun depmod. To then test initramfs-tools
> changes that skip over .zst compressed files and add them as is in an
> uncompressed appended cpio.

That is a very good idea. I created a draft for point 3 in [2]. It moves
the compressed files into a separate directory and creates a separate
cpio archive for that directory without compressing it:

* AMD/Intel microcode cpio archive (on amd64)
* main cpio archive (compressed)
* compressed kernel modules / firmware (not compressed)

Sadly this does not work (yet). cpio complains with "premature end of
archive" when looking at it and the kernel fails to extract the last
cpio part. I am heading to bed now leaving that bug for another day.


I had this before with lz4 and lzo compression and fixed this before in the kernel. It is likely a kernel bug mishandling mixed compression initrds.

It will likely work if you.... Make compressed-files the third early initrd. Such that we have intel, AMD microcode early initrds, followed by compressed-files initrd, followed by compressed main initrd.

Posted by at