IBM Security QRadar

In this tutorial, we will go over the need steps to get the Kubernetes API server logs into QRadar.

High level view on the steps:

1. to create an audit policy.
2. to configure Kubernetes api server logs to save the logs to a local file
3. to forward the logs from a local file to QRadar over Syslog.

First, we need to create the audit policy file, which defines which events to log or not, also using it we can determine the level of the details in the logged event.

Kubernetes offers the following audit levels:

None - don’t log events that match this rule.

Metadata - log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.

Request - log event metadata and request body but no response body.

Request Response - log event metadata, request, and response bodies.

There are several online ready Kubernetes audit policies, but some of them don’t cover the logging of some essential activities, like creating a role, adding a user to a role, or requests from unsecured Kubernetes API port

If we want to detect if someone will send a request to create a privileged pod, the HTTP metadata is not enough, we need to log the request body to see the request details, e.g., Pod name, Pod privilege level, ...

Of in some audit configs, they enable the role auditing at the metadata level, in such case, if someone will add a user to the cluster-admin role, it wont be detected, as the role name, and the added user name will not be included in the meta data.

Here you can find the audit policy that I have used on my cluster, and you can use it as a starting point.

Kubernetes offers several ways to export its logs, or push its API logs:

- Log backend, which writes events to a local file.

- Webhook backend, which buffers the events and send them to an external API

- Dynamic Webhook backend (AuditSink), which configures Kubernetes on the fly to send the events to a remote API.

Configuring the API server to log its output to a local file

In this tutorial we will cover the first option, to log the events to a local file, in several Kubernetes as a service, this option might be already enabled for you, as we have seen in this tutorial:

Ingesting Kubernetes Logs from Amazon Elastic Kubernetes Service (Amazon EKS)

But in case its not enabled, or you are using your own cluster, you can enable the local file logging by adding a few flags to the Kube apiserver config file, as follows (Note: you will need to SSH to every master node and to apply the following options):

vim /etc/kubernetes/manifests/kube-apiserver.yaml

1) Add the following entries under spec -> containers -> command > kube-apiserver

    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml

    - --audit-log-path=/var/log/kubernetes-apiserver.log

    - --audit-log-format=json

As the kube-apiserver will be running as a container, we need to share those two files: audit-policy.yaml, kubernetes-apiserver.log from the host to the container, which can be done using the next two steps.

2) Add the following entries under the volumeMounts:

    - mountPath: /etc/kubernetes/

      name: policies

      readOnly: false

    - mountPath: /var/log/

      name: logs

      readOnly: false

3) Add the following entries under volumes:

  - hostPath:

      path: /etc/kubernetes/

      type: DirectoryOrCreate

    name: policies

  - hostPath:

      path: /var/log/

      type: DirectoryOrCreate

    name: logs

After saving and exiting the config file, the Kube API server will be automatically restarted, but in case of any issues, you can restart it by restarting the kubelet

systemctl restart kubelet

To check if Kubernetes started to save the logs to the specified file, we will check if the file will have any new content:

head /var/log/kubernetes-apiserver.log

Forwarding the logs from a local file to QRadar

 
To forward the logs from a local file to QRadar, there are several open-source tools, we will cover the following solutions:

1. Using syslog-ng running as a DaemonSet
2. Using fluentd running as a DaemonSet
3. Using standalone syslog-ng

Using syslog-ng running as a DaemonSet

Here you will find a ready config file, named: syslog-ng-daemonset.yaml

Which will deploy syslog-ng as DaemonSet, and it will forward the logs from the following local file /var/log/kubernetes-apiserver.log to my QRadar IP: 192.168.0.45

You just need to modify the two parameters, the file path, and the QRadar IP.

After that, you can apply it to your cluster, as follows:

kubectl apply -f syslog-ng-daemonset.yaml

Using fluentd running as a DaemonSet

 

The needed fluentd config file is located in this collection, you just need to modify the file path and the QRadar IP, as follows:

To modify the file path:

To modify the QRadar IP:

Then, you can apply it to your Kubernetes cluster in a single command as:

kubectl apply -f fluentd-daemonset-syslog.yaml

Using standalone syslog-ng

 

You can use the standalone syslog-ng as well, in case if you don’t have it, you can install it as following:

wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04/Release.key | sudo apt-key add -

echo 'deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_18.04 ./' >> /etc/apt/sources.list

apt-get update && apt-get install -y syslog-ng-core && systemctl enable syslog-ng

To configure syslog-ng to forward the logs from a file, you can edit your own syslog-ng config file, or you can create a new one as follows:

nano /etc/syslog-ng/conf.d/kubelogs.conf

Add the following entries:

 

 

You will need to change the IP to your QRadar IP.

source s_kube_api_logs {file("/var/log/kubernetes-apiserver.log" flags(no-parse));};

destination d_qradar {network("192.168.0.45" transport("tcp") port(514));};

rewrite r_kubeAPI { set("kubernetesAPILogs_$HOST", value("HOST")); };

log {source(s_kube_api_logs); rewrite(r_kubeAPI); destination(d_qradar);};

 

After that, you just need to reload or restart syslog-ng:

systemctl reload syslog-ng

Kubernetes Logs in QRadar

 

QRadar will start to receive the logs, they will be auto discovered as Kubernetes, Security analysts can detect several threats targeting the Kubernetes cluster, like:

• Creation of a privileged container.
• Mounting of sensitive or critical volumes to a container.
• Failed requests to get the Kubernetes secrets.
• Leaked container token
• If a container token will be used to execute commands over other containers.
• If a container token will be used to create another container.
• Successful Kubernetes API requests from anonymous user.
• Successful Kubernetes API requests from unusual countries.
• Successful Kubernetes API requests from different geographies using the same access token.
• Multiple forbidden requests initiated from the same username.
• Command execution over containers in the Kubernetes system namespace.
• Creation of an unusual privileged role

 

The Kubernetes use cases are available out of the box and can be downloaded here: IBM QRadar Container Content Extension

All the config files referenced above can be downloaded from this IBM X-Force Exchange collection.