Hi all,
I thought I'd provide some context on this, reiterating what I wrote on Friday.
Summary: We would like to see some discussion about increasing the /boot partition for automatic partitioning on LVM/Encrypted systems to 2GB for 22.04. We believe this would be a relatively inexpensive change, and would be much better than the 1.5GB set to be a part of 20.04.4.
Background:
I'm part of the team that makes the Kubuntu Focus line of laptop computers. On encrypted LVM machines, using the default disk setup, we have found the /boot partition is way too small. We discovered that the /boot partition gets overfilled while apt is installing updates. Part of this problem is that Kubuntu uses Discover to do updating (not update-manager like other flavors excluding Ubuntu Studio which also uses Discover), which uses PackageKit as a backend.
PackageKit has a bug in which it individually installs packages when updating, causing those packages to be marked as manually installed and therefore not letting 'apt autoremove' do its job. As you can imagine, this breaks the old kernel removal process that update-manager does on other flavors. This was recently fixed upstream[1] in PackageKit and hopefully will release in time to beat the Feature Freeze / Debian Import Freeze.
We have created a script to mitigate this that automatically cleans kernels. We also increased the /boot partition size on our OEM-installed systems. We want to move our findings upstream to the OS. With the default 768MB /boot partition, sometimes even then three kernels would be installed and overfill with a new kernel update before it could be cleaned, especially if the user has more than one kernel flavor. A use case for two kernel flavors would be using 'lowlatency' for lower hardware latency, and using 'generic' for power-saving attributes when compared to 'lowlatency'. This would be especially advantageous on desktop-replacement-style laptop computers, such as the Kubuntu Focus M2[2].
Bear in mind also that the Nvidia drivers make the initrd for any given kernel significantly larger than it would be otherwise. With this in mind, we filed LP: #1960089[3], not knowing that Brian Murray had already patched and uploaded a larger boot partition prior [4]. Seeing Brian's patch, we don't believe it still makes the /boot partition large enough, especially considering the Nvidia drivers.
From LP: #1960089 (sizes here reported in MiB and GiB):
Summary:
We propose to increase the LVM /boot partition to 2.0 GB. This provides the space needed so advanced users can use best practice to manage up to 3 kernel flavors. The current /boot partition on 20.04 and 22.04 is limited to just 705M, which allows only 3 concurrent kernels before filling and sometimes locking the system (each image set takes 180M total; 4 x 180 = 720M > 705M).
Reasoning:
Best practice recommends users keep at least two version of each kernel flavor in the /boot directory. If a user has 3 kernel flavors installed (e.g. oem, generic-hwe, and lowlatency-hwe), then one needs to reserve room for 2 x 3 = 6 kernels.
The system needs the headroom of at least two additional kernels during any automated clean-up process due to package removal scheduling. I propose to also reserve room for 2 additional kernels as a safety measure. Thus the total recommend available space should accommodate 10 kernels.
Each kernel file set takes up 180M in the /boot partition when used with Nvidia driver modules. These files include initrd.img, system.map, and vmlinuz. With future kernel and module growth, this may surpass 200M soon. Therefore, we suggest planning for 200M for each kernel.
We therefore request a total LVM /boot partition size of 10 image x 200M = 2.0 GB.
Other Considerations:
When unattended-upgrades works correctly (which does not yet employ best practice), we have seen users with just a single kernel flavor over-fill their /boot partitions. This is because unattended-upgrades can retain up to 4 kernels, while the /boot partition is only large enough for 3. I am currently working with others to improve the unattended-upgrades algorithm to use best practice.
The installer could allow users to resize the /boot partition during installation. In this case, we highly recommend a 2.0 GB default for the reasons outlined above.
After discovering LP: #1959971, Mike commented:
Hi Łukasz and Brian: I have been doing quite a bit of research in this area recently, and also am advocating to get kernel management and cleanup improved, especially for users who must have separate /boot partitions. This means professional users who are required to have full disk encryption according to company IT policies.
Using best practice to manage 3 kernel packages (e.g. oem, lowlatency-hwe, generic-hwe) results in the need to maintain 6 kernel file sets (latest and penultimate version for each kernel). Because of the way kernel cleaning is scheduled, additional space is also required for at least 2 additional images. You can see the entire reasoning and details at LP: #1960089.
Our conclusion is that 2.0 GB is the preferred target /boot partition size for professional systems. This is reinforced by my research which shows many pros recommend this same size when discussing partitioning (Just one example: https://adventures-in-tech.blogspot.com/2018/10/encrypted-ubuntu-installation-with.html )
Given disk space these days, this seems like a small price to pay to support best practice and (hopefully) a future unattended upgrades algorithm that honors them as well.
I happened to see some discussion in #ubuntu-meeting by the Foundations Team about this, mentioning Mike's calculations are off. We were told to respond via this list, at which point Mike wrote the following (which got moderated as he's not an Ubuntu Developer per se):
On Thu, 2022-02-10 at 18:38 -0800, Michael Mikowski wrote:
Hi Everyone:Members of the ubuntu-meeting asked me to clarify the issue with /boot. You can see the bugs filed at https://launchpad.net/bugs/1959971 and https://launchpad.net/bugs/1960089.
Personal Background:I currently lead the Kubuntu Focus engineering efforts, and have been a professional product engineer specializing in mission-critical HP/HA/HV systems for decades. You can see a overview of my experience at https://michaelmikowski.com.My Assessments:0. This is a low-cost, high-return proposal. People have pushed back against increasing the /boot partition, but I find most of the reasoning does not consider the true impact of a small /boot partition to the complete product.1. The cost to provide the space needed is minimal for almost all FDE systems. So why not just do it? Of course, there is more work to be done for the rest of the product (guide users on kernel selection; improve the kernel cleaning logic), but this is an important component that is required in any complete solution. It would provide substantial relief to many users immediately.2. An overfull /boot partition is catastrophic for many users. Many cannot recover their system because they don't have a rescue disk or knowledge of ext4, chroot, LUKS, lvm, cryptesetup, package management, and more. These people either reload the OS or give up.3. This happens frequently, even when everything works as designed, and even when just using a single kernel flavor. While on IRC yesterday discussing this, 3 unsolicited individuals stepped in to comment about how this is needed. And those are advanced developers who know better. Search for 'ubuntu full /boot partition' and check out how frequently this continues to occur.4. There are many use cases where multiple kernel flavors will occur, such as when the users switches from HWE to OEM to address hardware issues, or they install lowlatency for studio work. When this happens, the current size boot partition is highly likely to fill. This can corrupt the system and require a rescue disk for recovery.Check out the DFMEA which is used to assess the severity of a failure mode: https://bugs.launchpad.net/ubuntu/+source/partman-auto/+bug/1959971/comments/7Finally, I assure you the calculations for kernel boot-file-set sizes (180M with Nvidia GPU) are correct for 20.04, and will be correct for 22.04 unless the compression technique has been changed. It seems perfectly reasonable to expect this size to grow to 200M or more over the next 2 years. Also, the headroom + safety factor specified (10 kernel file-sets total) is reasonable when you consider that systems that reboot less frequently will accumulate kernels and that a single upgrade can install multiple additional kernel images.Sincerely, Mike
Łukasz Zemczak responded to LP: #1959971 today following a detailed analysis Mike did in Comment #7.
Thank you for the detailed assessment! For 20.04 (at least for 20.04.4) we will only bump the partition sizes slightly, as mentioned in the current description (and in the current SRU). We need to thread carefully as it's a point-release of an already released series. As for 22.04, we are certainly open to discussion and your findings will be very helpful to make the best decision.
Let's see how it goes.
We are in 100% agreement with Łukasz on this and would like to see some discussion on this increase in /boot partition size.
Thanks in advance,
Erich
--
Erich Eickmeyer
Project Leader, Ubuntu Studio
Member, Ubuntu Community Council
UX & Packaging Director, Kubuntu Focus
