Wednesday, 11 January 2017

Re: Make systemd journal persistent | remove rsyslog (by default)

> From: [email protected] [mailto:ubuntu-devel-
> Jamie Strandboge [2017-01-10 16:27 -0600]:
> > Remote logging. Rsyslog is far superior in this regard. Granted,
> remote logging
> > is not enabled by default but it is a requirement in many
> environments.
> The systemd-journal-remote package does provide the necessary tools and is
> reasonably flexible (push or pull, builtin https or using arbitrary ports
> which
> you e. g. could forward through ssh). It might not be as flexible as
> rsyslog, but as one needs to set up remote logging manually anyway, you
> always
> have the possibility of picking rsyslog, journal, or even something else.

I have done a POC with systemd-journal-remote yet, but that sounds quite good.
In our production environment following features should be supported also by
Rsyslog replacements to support current usecases and ease gradual rollout:

1) rsyslog RELP mode (reliable remote transport) - reduces probability for
lost messages during network maintenance, e.g. statefull firewall restarts
without conntrack failover.

2) Remote log data caching on network downtimes - data should be transmitted
when network up again

3) Cascading operation: branch offices or guest perform remote logging to a
nearby concentrator, only the concentrator has to be granted off-site access
in firewall

4) TLS support with own CA - each new machine will get a new certificate from
deployment system, central logging should accept traffic from all those
machines with valid certificate automatically (eases automatic rollout)

5) rsyslog/system hybrids: in early phase there will be e.g. some old rsyslog
based LXC containers forwarding logs to systemd-only hosts and perhaps also
vice versa.

6) Streams > 10GB logs/day should not cause any troubles, log data losses.

7) Integration with SIEMs or logmanagement solutions, e.g. graylog.

Perhaps not all those features are possible/sensible to have
systemd-journal-remote. In that case another requirement could arise:

8) Allow local rsyslog/systemd-journal-remote combination: systemd forwards
the logs to a local rsyslog (which might also process other remote data from
requirement 3), and all the really remote forwarding will happen using
old-style rsyslog. Of course this would require maintenance of automation
scripts for setup/automatic configuration of both rsyslog/systemd in various
flavours (host/LXC-guest, Trusty/Xenial/CentOS)

I put it on my list, that I have to do a ~3 machine POC with
systemd-journal-remote only setup to check all those requirements. As rsyslog
is not completely trouble-free, this might help cut down costs in the long