Has anyone built their own solution to get the lightweight solutions logging to a remote syslog server, outside of the container orchestration's capability of doing this?  The reason I ask is that I believe we are going to be forced onto the lightweight containers soon (I have a case open with L2, but we cannot get the v10.0.3.0 runtime to start on the full image).  We are not ready to configure our orchestration environment for sending the console logs to syslog, and then train our log correlation software to ingest the new JSON log formats.  For anyone else's reading, if you're not aware, the lightweight containers drop the rsyslog forwarder feature and send all logs to stdout to the container console log.

I was thinking of building a sidecar app to each ISVA pod that would run rsyslog or to build an app that runs it and monitors the shared storage (PVC).  In any case, either would tail the logs and send to the syslog daemon local to that app.

Has anyone done anything like this?  If so, are there any examples to share?

Thanks for any insight.

------------------------------
Matt
------------------------------

Hello,

Without wanting to sound opinionated, there are a couple of lightweight agents that can run as sidecar . 
https://fluentbit.io/blog/2020/12/03/common-architecture-patterns-with-fluentd-and-fluent-bit/
Syslog as output : https://docs.fluentbit.io/manual/pipeline/outputs/syslog
A couple of examples
https://github.com/StevenACoffman/fluent-bit-tomcat-sidecar
https://github.com/leahnp/fluentbit-sidecar

Also have a look at https://vector.dev/docs/setup/deployment/roles/#sidecar.  Vector.dev can very straightforward be  configured with File (as source) and choice of many sinks .  You can also transform (Vector remap language) the data between source and sink.  The solution is very lightweight .
This solution is similar to Fluentbit or Fluent agents ...  There is also the option to read the Kubernetes logs (https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/)  and use pod_annotation_fields which allows you to filter out the logs from your ISVA (specific ) pods.

Hope this helps
Kind regards
Serge Vereecke
IBM Security

------------------------------
Serge Vereecke
------------------------------

Original Message:
Sent: Mon February 07, 2022 04:20 PM
From: Matt Jenkins
Subject: Lightweight containers forward logs to syslog

Has anyone built their own solution to get the lightweight solutions logging to a remote syslog server, outside of the container orchestration's capability of doing this?  The reason I ask is that I believe we are going to be forced onto the lightweight containers soon (I have a case open with L2, but we cannot get the v10.0.3.0 runtime to start on the full image).  We are not ready to configure our orchestration environment for sending the console logs to syslog, and then train our log correlation software to ingest the new JSON log formats.  For anyone else's reading, if you're not aware, the lightweight containers drop the rsyslog forwarder feature and send all logs to stdout to the container console log.

I was thinking of building a sidecar app to each ISVA pod that would run rsyslog or to build an app that runs it and monitors the shared storage (PVC).  In any case, either would tail the logs and send to the syslog daemon local to that app.

Has anyone done anything like this?  If so, are there any examples to share?

Thanks for any insight.

------------------------------
Matt
------------------------------