Friday 13 January 2017

AW: Make systemd journal persistent | remove rsyslog (by default)

> From: ubuntu-devel-bounces@lists.ubuntu.com [mailto:ubuntu-devel-
> bounces@lists.ubuntu.com] Im Auftrag von Jamie Strandboge
> Subject: Re: Make systemd journal persistent | remove rsyslog (by
> default)
>
> On Thu, 2017-01-12 at 10:50 -0500, Bryan Quigley wrote:
> > We could explicitly keep rsyslog supported in main for at least 18.04
> > for the for those who need it (or indefinitely if we find it's still
> > needed for remote enterprise logging). I was thinking that we might
> > have to keep it in main until 18.04 anyway for upgrades.
> >
> I think this would be a hard requirement if it was decided on the
> switch.
>
> Another thing that came to mind is 'logcheck' (in main) for log auditing
> and I
> don't think it understands systemd-journald log format. logcheck is not
> installed by default of course, but it is another package useful in
> enterprise
> environments. If the standard logs are removed, then installing logcheck
> won't
> work by default and additional steps need to be performed to install
> rsyslog
> (and make sure systemd-journald forwards to it).
>
> There are two things here:
> 1. make systemd journal persistent
> 2. avoid duplicate logs from rsyslog
>
> Why not just do '1' and let rsyslog remain? The standard logs are
> rotated so
> this shouldn't be overly burdensome. Have you measured how much the
> duplicate
> logs would take on a typical system?

Why not try building an improved logcheck supporting sockets/streams? We also
used logcheck in our environment, but in one of our projects we experimented
with a more configurable variant, that could overcome typical problems seen
with logcheck, e.g. run also on small devices (logcheck in batch needs large
amount of disk/memory to create large reports), allow more finegrained
whitelisting, custom filters/detectors/analyzers, stream operation, fast
mail/custom alerting (e.g. 10sec after first event with logarithmic backoff),
... We now use it also on some of our internal machines. We also published the
IO and primitive filtering layer open source as "logdata-anomaly-miner".
Currently it is still hell to configure, but perhaps with some effort writing
good "default check configurations", systemd with UNIX socket and stream
logchecking might be interesting. At the moment we use the socket scheme to
connect AMiner to auditd, where the amount of log data to analyze might be too
large to write it to disk or send it to remote filtering machine.

Perhaps our approach is too much on the research/experimentation side, but
logdata-anomaly-miner IO reading/alerting + logcheck regex module might be
easy to build and even give backward compatibility with logcheck/operation on
readonly root disk systems, ...