Tuesday, 7 November 2017

Should Ubuntu systemd journal logs be persistent by default?

Some time around the 15.04 release, a policy change was made to quit making some logging persistent by default.

A number of users did not realize there was a policy change until they went to debug something that happened on a previous boot, only to find the logs were missing. Some of these user responses are captured in the bug report below [1], which prompted this policy discussion.

The change was made because of the introduction of systemd and the introduction of the `systemd journal` in addition to the existing `rsyslog`.

The concern about making the systemd journal persistent by default is that some logs could end up duplicated between the systemd journald and rsyslog, along with disk space and performance concerns of the additional logging. 

On the other hand, having systemd journal logs persist seems to be a safer option: It a malicious app could cause or entice a reboot, it could erase logs of it's earlier activity. Also, deleting key logs at shutdown breaks a decades-log precedent of  system logs persisting through reboots. 

The compromise option seems to be make the systemd journal persistent by default, but minimize the amount of  logging that is sent to both rsyslog and systemd to mitigate resource considerations of duplicate logging.

So the policy question here is: should the systemd journal be persistent by default?