Monday, 13 November 2017

Re: Should Ubuntu systemd journal logs be persistent by default?



On Mon, Nov 13, 2017 at 2:27 AM, Robie Basak <[email protected]> wrote:
Hi Mark,

On Tue, Nov 07, 2017 at 08:11:27PM +0000, Mark Stosberg wrote:
> Some time around the 15.04 release, a policy change was made to quit
> making some logging persistent by default.

To help me understand the problem, exactly what logging was persistent
previously, and isn't persistent now? Can you please give us an example?

Any data in systemd-journal that is not flush to  /var/log/* is gone forever on power fail due
to the journal being hosted on ephemeral tmpfs (/run/log/journal); 

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
[email protected]:/var/log# ls -al /var/log/
total 500
drwxrwxr-x  8 root      syslog   4096 Nov 13 15:19 .
drwxr-xr-x 12 root      root     4096 Nov 10 20:43 ..
drwxr-xr-x  2 root      root     4096 Nov 10 20:43 apt
-rw-r-----  1 syslog    adm      2884 Nov 13 15:19 auth.log
-rw-r--r--  1 root      root     7678 Nov 13 15:12 boot.log
-rw-rw----  1 root      utmp        0 Nov 10 20:43 btmp
-rw-r--r--  1 syslog    adm    153678 Nov 13 15:12 cloud-init.log
-rw-r--r--  1 root      root     3870 Nov 13 15:12 cloud-init-output.log
drwxr-xr-x  2 root      root     4096 Nov 30  2016 dist-upgrade
-rw-r--r--  1 root      adm     30483 Nov 13 15:12 dmesg
-rw-r--r--  1 root      root        0 Nov 13 15:12 dmesg.0
drwxr-xr-x  2 root      root     4096 Nov 10 20:41 fsck
-rw-r-----  1 syslog    adm     52797 Nov 13 15:13 kern.log
drwxr-xr-x  2 landscape root     4096 Nov 13 15:12 landscape
-rw-rw-r--  1 root      utmp   292292 Nov 13 15:19 lastlog
-rw-r-----  1 syslog    adm     55736 Nov 13 15:17 syslog
-rw-r--r--  1 root      root   148851 Nov 13 15:12 udev
drwxr-xr-x  2 root      root     4096 May  8  2017 unattended-upgrades
drwxr-xr-x  2 root      root     4096 Nov 13 15:19 upstart
-rw-rw-r--  1 root      utmp     5376 Nov 13 15:19 wtmp


$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
[email protected]:/var/log$ ls -al /var/log/
total 312
drwxrwxr-x  7 root   syslog   4096 Nov 13 15:15 .
drwxr-xr-x 13 root   root     4096 Nov 10 14:25 ..
drwxr-xr-x  2 root   root     4096 Nov 10 14:24 apt
-rw-r-----  1 syslog adm      3181 Nov 13 15:17 auth.log
-rw-------  1 root   utmp        0 Nov 10 14:24 btmp
-rw-r--r--  1 syslog adm    127616 Nov 13 15:15 cloud-init.log
-rw-r--r--  1 root   root     4271 Nov 13 15:15 cloud-init-output.log
drwxr-xr-x  2 root   root     4096 Oct 20 10:35 dist-upgrade
drwxr-xr-x  2 root   root     4096 Nov 10 14:22 fsck
-rw-r-----  1 syslog adm     52555 Nov 13 15:15 kern.log
-rw-rw-r--  1 root   utmp   292292 Nov 13 15:16 lastlog
drwxr-xr-x  2 root   root     4096 Aug 23 02:06 lxd
-rw-r-----  1 syslog adm     82360 Nov 13 15:17 syslog
drwxr-x---  2 root   adm      4096 Sep 20 14:13 unattended-upgrades
-rw-rw-r--  1 root   utmp     2688 Nov 13 15:16 wtmp

boot.log includes services that start on boot, 

These service starts are not part of syslog;  ssh for example in boot.log shows
it starting (but any service may include more information here in case of failure).

# grep -i openssh boot.log 
 * Starting OpenSSH server                                               [ OK ]

$ journalctl -o short-precise --no-pager -u ssh
-- Logs begin at Mon 2017-11-13 15:15:13 UTC, end at Mon 2017-11-13 15:17:01 UTC. --
Nov 13 15:15:33.816052 x-undiluvial-edith systemd[1]: Starting OpenBSD Secure Shell server...
Nov 13 15:15:33.846646 x-undiluvial-edith sshd[1249]: Server listening on 0.0.0.0 port 22.
Nov 13 15:15:33.847790 x-undiluvial-edith sshd[1249]: Server listening on :: port 22.
Nov 13 15:15:33.848100 x-undiluvial-edith systemd[1]: Started OpenBSD Secure Shell server.
Nov 13 15:16:35.015383 x-undiluvial-edith sshd[1317]: Accepted publickey for ubuntu from 10.5.0.10 port 54030 ssh2: RSA SHA256:0cuEJGbgH9aUy12t0jKluNEwLeAF4SIbvimjeM1OLWw
Nov 13 15:16:35.018911 x-undiluvial-edith sshd[1317]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

/var/log/udev is another.

In general when systems fail to boot, users may not have console access but may retain power control.
Issuing a shutdown or even hard stop will still preserve data that's been written to the filesystem for later
inspection.  With the bulk if boot and service status and output information included in the journald held in
tmpfs; we lose that data unless there is some way to flush that data to persistent storage.

I'm +1 for having journald be persistent by default; it certainly makes debugging boot failures
much easier since one has actual logs to examine versus nothing at all.


Thanks,

Robie

--
ubuntu-devel mailing list
[email protected]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel