Re: Adjusting what fstypes df displays

On Fri, May 29, 2020 at 10:54:40PM +0000, Seth Arnold wrote:
> On Thu, May 28, 2020 at 09:38:47PM -0700, Bryce Harrington wrote:
> > For Ubuntu, DF_EXCLUDE_FSTYPES could be set in the global profile (under
> > /etc/profile.d/ perhaps?) This would make it straightforward to add
>
> Many thanks for working on this, df output has been annoying me for quite
> some time (but I dislike shell aliases even more).
>
> Properly threading an environment variable into a given process is
> surprisingly difficult. /etc/profile.d/ is read by bash, and other
> shells that choose to use these startup files. /etc/environment is
> used by any PAM service that has pam_environment configured and
> systemd-environment-d-generator(8) for systemd user manager instances.
>
> These may not cover all instances where users would like it to. And,
> ironically, even though it doesn't cover everything, it would be in the
> memory of many processes that may never call df. It's not like it's a
> huge amount of memory, but it is something.
>
> Different tools like sudo, su, systemd-nspawn, lxc exec, etc may
> further complicate getting an environment variable through from where a
> human is interacting with the system to where df is executing.
>
> So, I'd like to propose using the typical systemd-style configuration file
> approach instead:
>
> Package defaults in:
> /lib/something/coreutils/df.conf or
> /usr/lib/something/coreutils/df.conf
>
> Sysadmin configuration in:
> /etc/something/coreutils/df.conf
>
> User configuration in:
> XDG_SOMETHING_CONFIG/coreutils/df.conf
> or
> /run/uid/something/coreutils/df.conf
>
> Yes, it's exhausting to implement the functionality for all this compared
> to the simplicity of a getenv() or secure_getenv(), but filesystem access
> is more predictable than environment variables from parents to children
> and will likely work closer to expectations across chroot, schroot, lxd,
> etc uses.
>
> If we ever want to change the package defaults, then any local admin
> choices already made could continue to be respected.
>
> Thanks

Thanks for the feedback Seth.

I haven't seen other examples of config file use elsewhere in coreutils,
which was one reason I went with an env var for this POC.

I'll bring your points forward when I present to upstream, and if they
also like the conf file approach better, I can reimplement it that way.
Is there another GNU package (in C) that implements a systemd-style
config file approach that you'd recommend to use for reference?

Bryce

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Adjusting what fstypes df displays

On Fri, May 29, 2020 at 01:58:34PM -0700, Bryce Harrington wrote:
> This is probably too low-level to be worth including in the Groovy
> release notes, but I can propose a sentence or two just for some extra
> visibility.

IMHO, if it happens, it's worth putting in the release notes.

Re: Adjusting what fstypes df displays

On Fri, May 29, 2020 at 10:31:56AM -0400, Sergio Durigan Junior wrote:
> On Friday, May 29 2020, Bryce Harrington wrote:
> > I've drafted a POC implementation for df here:
> >
> > https://github.com/bryceharrington/coreutils/commit/cc372d778b44abcf81af42743a5174685f9bb4db
>
> Hey Bryce,
>
> Overall I like the idea of using an environment variable to control this
> behaviour, but I feel compelled to raise the point of discoverability.
>
> If the distro ships a file under /etc/profile.d/ which silently sets
> this variable for all users, and if the user is not aware of that, it
> may take her some non-trivial effort to figure out why her tmpfs mount
> is being listed by df.
>
> I'm sure you must have thought of that, but if you go forward with the
> environment variable idea, extra care will need to be taken when writing
> the documentation for it. Perhaps (and that's a big "perhaps"; I
> haven't thought much about it) it's even worth mentioning somehow that
> DF_EXCLUDE_FSTYPES is being used in the output of df,

It's a good point. My plan is to mention it in --help, and also add an
ENVIRONMENT section to the df man page, since at least to me that'd be
the second and third places I'd look (the first being google
obviously...) I notice that df has a couple other environment variables
(POSIXLY_CORRECT and DF_BLOCK_SIZE) mentioned in --help but not the man
page, which could go in as well.

This is probably too low-level to be worth including in the Groovy
release notes, but I can propose a sentence or two just for some extra
visibility.

> or having a new
> "--full-output" option (or some such) that would ignore the variable and
> print everything.

There is a 'df --all' option that sounds like it should suit that need.
Thanks for raising this idea -- in testing it I notice the new code
isn't honoring it. I'll add a check to skip the env var if -a,--all is
specified.

> I kind of like the idea of a configuration file, but I agree that an
> environment variable is more flexible. Also, I don't think the creation
> of a configuration file would be justified for just this single option.

Yeah, those were my thoughts as well. df itself doesn't currently use a
config file, but I wonder if coreutils itself has something.

> Let me know when you submit the patch upstream; I'm interested in
> following the discussions :-).

Sure thing,
Bryce

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Adjusting what fstypes df displays

On Fri, May 29, 2020 at 11:15:27AM -0300, Rafael David Tinoco wrote:
> On 29/05/2020 11:03, Robie Basak wrote:
> > +1 for Bryce's approach.
> >
> > On Fri, May 29, 2020 at 10:56:34AM -0300, Rafael David Tinoco wrote:
> >> Perhaps this environment variable should be set by snapd package ?
> > I think this could be surprising - because we would be hiding all
> > squashfs filesystems from df, not just snapd ones. I think it would be
> > cleaner to consider that users don't generally want to see squashfs
> > stuff in df output by default anyway, and so hide it Ubuntu-wide.
> >
> > I'm less sure about tmpfs. I can think of occasions where I have wanted
> > to see tmpfs usage (since it can run out, and has run out on me before).
> > But that's perhaps too much of a specialist case, and so I think it's
> > also OK to hide tmpfs by default from df on Ubuntu.
> I'm wondering if having an optarg to disconsider the environment
> variable would be appropriate. And sorry if that was already considered
> Bryce, I haven't looked into your code.

I didn't include a cli arg, but that effect can be achieved by:

$ DF_EXCLUDE_FSTYPES= df

I did make sure the env var plays well with -t, so if you *just* want to
see tmpfs and squashfs filesystems, for instance, then this would do it:

$ df -t tmpfs -t squashfs

(Note that if you've aliased df to 'df -x tmpfs' this will break if you
try to also -t tmpfs; those options are mutually exclusive, and df will
error.)

You can also get df to display everything, including normally hidden
filesystems via:

$ df -a

For my desktop this shows 79 entries, including cgroup and nsfs stuff
(that apparently snap uses) which aren't shown by default. The -T flag
is handy to use with -a to help figure out what's what.

$ df -a -T

Thanks for the feedback,
Bryce

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Adjusting what fstypes df displays

On Fri, May 29, 2020 at 12:39 AM Bryce Harrington
<bryce.harrington@canonical.com> wrote:
>
> These days, df displays a lot of mount points, due to the increased use
> of non-consumable filesystems such as tmpfs and squashfs. This clutter
> is particularly noticeable using df in Ubuntu, due to the increased
> popularity of snaps, but the general problem affects all Linux distros,
> and shows up in other commands such as lsblk, blkid, fdisk -l, and
> mount.
>
> A commonly documented user customization[1] is to use aliases, e.g.:
>
> alias df='df -h -x squashfs -x tmpfs -x devtmpfs'
> alias lsblk='lsblk -e 7'
>
> df's upstream is aware of the issue, and would like to address it in
> their code. They've considered a number of solutions[2], but do not
> appear to have come to a consensus as of yet. Among the raised concerns
> are variations in distro requirements, and providing an ability for
> users to override/customize the exclusions.
>
> A solution I am considering to propose would add an environment
> variable, DF_EXCLUDE_FSTYPES, that df would treat similar to the -x
> flag.
>
>
> I've drafted a POC implementation for df here:
>
> https://github.com/bryceharrington/coreutils/commit/cc372d778b44abcf81af42743a5174685f9bb4db
>
>
> Here's what it does inside a container...
>
> Before:
>
> $ df
> Filesystem 1K-blocks Used Available Use% Mounted on
> default/containers/coreutils 24891008 2258816 22632192 10% /
> none 492 4 488 1% /dev
> udev 32862724 0 32862724 0% /dev/tty
> tmpfs 100 0 100 0% /dev/lxd
> /dev/nvme0n1p2 982940092 348781368 584158392 38% /home/bryce/pkg
> tmpfs 100 0 100 0% /dev/.lxd-mounts
> tmpfs 32892912 0 32892912 0% /dev/shm
> tmpfs 6578584 224 6578360 1% /run
> tmpfs 5120 0 5120 0% /run/lock
> tmpfs 32892912 0 32892912 0% /sys/fs/cgroup
> tmpfs 6578580 0 6578580 0% /run/user/1001
>
> After:
>
> $ export DF_EXCLUDE_FSTYPES=tmpfs,devtmpfs,squashfs
>
> $ df
> Filesystem 1K-blocks Used Available Use% Mounted on
> default/containers/coreutils 24891008 2258816 22632192 10% /
> /dev/nvme0n1p2 982940092 348781372 584158388 38% /home/bryce/pkg
>
> (It's even more drastic on my desktop - from 39 lines down to 6.)
>
> For Ubuntu, DF_EXCLUDE_FSTYPES could be set in the global profile (under
> /etc/profile.d/ perhaps?) This would make it straightforward to add
> new non-consumable filesystems that may crop up down the road. Perhaps
> other tools could use a similar/same env var approach, giving us a
> uniform way to control fs listings in the distro.
>
> Two alternate approaches I'm weighing are a config file in /etc, or just
> a package patch to carry in coreutils, but the env var approach feels
> like it would be more flexible to users and hopefully more suitable for
> upstream. Before I forward it upstream, though, what does ubuntu-devel@
> think?

I agree on the pain of df output pollution, for sure. And patching df
would help; though as you mentioned it's not the only tool affected,
e.g. lsblk, and IMO mount output is even worse.

While I'm +1 on adding an easy way to hide them, I'm -1 on hiding all
squashfs and tmpfs from everyone *by default*. I think that will lead
to a lot of confusion. But that's just my opinion.

Also re: using an env var, keep in mind there are quite a few
environments that won't keep profile vars, e.g. sudo, lxc exec, etc.
It could be quite confusing for 'df' to show one thing, and 'sudo df'
something else.

>
> Thanks,
> Bryce
>
>
>
> 1: Google shows many links, here's a sample:
>
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1756595
> https://askubuntu.com/questions/1029493/how-can-i-stop-snaps-from-listing-in-df
> https://clintonminton.com/how-to-clean-up-snap-and-loop-devices-from-df-output-in-ubuntu-18-04/
>
> Note that one drawback to this approach is it's not overrideable, i.e.:
>
> $ alias df='df -h -x squashfs -x tmpfs -x devtmpfs'
> $ df -t tmpfs
> df: file system type 'tmpfs' both selected and excluded
>
> 2: Ideas the maintainers discussed include omitting read-only devices,
> or ones with non-consumable storage, or with <1% usage. They also
> discussed maintaining a simple list of excludable "lesser" file
> systems, possibly stored in a user's ~/.dfrc or ~/.shellrc.
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37702
>
> 3: We exclude squashfs,tmpfs,devtmpfs in our nagios packaging, where
> these were generating false-positive 'disk full' alerts:
>
> https://bugs.launchpad.net/charm-nagios/+bug/1827159
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Adjusting what fstypes df displays

On Friday, May 29 2020, Bryce Harrington wrote:

> These days, df displays a lot of mount points, due to the increased use
> of non-consumable filesystems such as tmpfs and squashfs. This clutter
> is particularly noticeable using df in Ubuntu, due to the increased
> popularity of snaps, but the general problem affects all Linux distros,
> and shows up in other commands such as lsblk, blkid, fdisk -l, and
> mount.
>
> A commonly documented user customization[1] is to use aliases, e.g.:
>
> alias df='df -h -x squashfs -x tmpfs -x devtmpfs'
> alias lsblk='lsblk -e 7'
>
> df's upstream is aware of the issue, and would like to address it in
> their code. They've considered a number of solutions[2], but do not
> appear to have come to a consensus as of yet. Among the raised concerns
> are variations in distro requirements, and providing an ability for
> users to override/customize the exclusions.
>
> A solution I am considering to propose would add an environment
> variable, DF_EXCLUDE_FSTYPES, that df would treat similar to the -x
> flag.
>
>
> I've drafted a POC implementation for df here:
>
> https://github.com/bryceharrington/coreutils/commit/cc372d778b44abcf81af42743a5174685f9bb4db

Hey Bryce,

Overall I like the idea of using an environment variable to control this
behaviour, but I feel compelled to raise the point of discoverability.

If the distro ships a file under /etc/profile.d/ which silently sets
this variable for all users, and if the user is not aware of that, it
may take her some non-trivial effort to figure out why her tmpfs mount
is being listed by df.

I'm sure you must have thought of that, but if you go forward with the
environment variable idea, extra care will need to be taken when writing
the documentation for it. Perhaps (and that's a big "perhaps"; I
haven't thought much about it) it's even worth mentioning somehow that
DF_EXCLUDE_FSTYPES is being used in the output of df, or having a new
"--full-output" option (or some such) that would ignore the variable and
print everything.

> Here's what it does inside a container...
>
> Before:
>
> $ df
> Filesystem 1K-blocks Used Available Use% Mounted on
> default/containers/coreutils 24891008 2258816 22632192 10% /
> none 492 4 488 1% /dev
> udev 32862724 0 32862724 0% /dev/tty
> tmpfs 100 0 100 0% /dev/lxd
> /dev/nvme0n1p2 982940092 348781368 584158392 38% /home/bryce/pkg
> tmpfs 100 0 100 0% /dev/.lxd-mounts
> tmpfs 32892912 0 32892912 0% /dev/shm
> tmpfs 6578584 224 6578360 1% /run
> tmpfs 5120 0 5120 0% /run/lock
> tmpfs 32892912 0 32892912 0% /sys/fs/cgroup
> tmpfs 6578580 0 6578580 0% /run/user/1001
>
> After:
>
> $ export DF_EXCLUDE_FSTYPES=tmpfs,devtmpfs,squashfs
>
> $ df
> Filesystem 1K-blocks Used Available Use% Mounted on
> default/containers/coreutils 24891008 2258816 22632192 10% /
> /dev/nvme0n1p2 982940092 348781372 584158388 38% /home/bryce/pkg
>
> (It's even more drastic on my desktop - from 39 lines down to 6.)
>
> For Ubuntu, DF_EXCLUDE_FSTYPES could be set in the global profile (under
> /etc/profile.d/ perhaps?) This would make it straightforward to add
> new non-consumable filesystems that may crop up down the road. Perhaps
> other tools could use a similar/same env var approach, giving us a
> uniform way to control fs listings in the distro.
>
> Two alternate approaches I'm weighing are a config file in /etc, or just
> a package patch to carry in coreutils, but the env var approach feels
> like it would be more flexible to users and hopefully more suitable for
> upstream. Before I forward it upstream, though, what does ubuntu-devel@
> think?

I kind of like the idea of a configuration file, but I agree that an
environment variable is more flexible. Also, I don't think the creation
of a configuration file would be justified for just this single option.

Let me know when you submit the patch upstream; I'm interested in
following the discussions :-).

Thanks,

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Upcoming git-ubuntu repository disruptions

Now that 1.0-rc1 is released, the importer output for the "unapplied"
branches and tags is now stable and I don't expect any further changes.
To bring the existing experimental repositories out of experimental
status, I'm going to "reimport" everything once.

This will cause all commits and tags to "change hashes", leading to
non-fast-forwarding updates to all branches. If you are following the
experimental respositories, you will need to accommodate - perhaps by
rebasing any existing work on the new imports.

Once all repositories are updated, we will be able to take them out of
"experimental" status and they will be able to be considered stable for
regular use.

Note that this only applies to the "unapplied" branches and tags.
"applied" branches and tags and the various support refs such as for
pristine-tar and dsc files may still non-fast-forward at any time.
However I don't expect that to be a problem for how the git-ubuntu
repositories are typically used today.

Thanks,

Robie