IBM Cloud Global

 View Only

Enhancing Kubernetes cluster security through IBM Workload Protection and Kubernetes audit logs

By Ahcene fekir posted Mon July 29, 2024 03:27 AM

  

In today’s fast-evolving technology, developers and DevOps teams are creating, releasing, and deploying applications faster than ever. Kubernetes clusters have become essential for managing these containerized applications efficiently. However, as the pace of development accelerates, ensuring the safety and security of these environments become critical. It is imperative to keep up with these rapid changes and quickly identify any anomalies.

This article explores the importance of Kubernetes audit logs, and the critical role of IBM Cloud Security Compliance Center Workload Protection in safeguarding. Additionally, we'll introduce Falco, a cloud-native security tool. 

These audit logs, Falco and Workload Protection will create a strong security shield for your Kubernetes world.

Kubernetes audit logging

The Kubernetes audit logging is a detailed record of cluster activities. It captures information about user interactions, API requests, system changes, and resources access. It provides administrators a precise status and accounting of cluster actions.

What is Falco

Falco is a guardian tool that actively monitors your kubernetes cluster and detects anything suspicious in real-time, It understands security activities, generates alerts or takes action when predefined rules are violated, it analyzes log data to find abnormal behaviors and potential threats. It ensures that all suspicious activities are promptly identified and addressed.

The benefits of Kubernetes audit logs

Securing and maintaining reliable Kubernetes environments is crucial for business using containerized applications. By combining these audit logs, administrators gain the ability to address fundamental questions: 

 

  • What happened? An attempt was made to delete a critical namespace within the kubernetes cluster.

  • Who initiated the action? The action was initiated by a user with the identity toto.tata@ibm.com

  • When did it happen? The deletion attempt occurred on March 11,2025 at 15:45 UTC

  • Where did it occur?  The action originated from the IP address 10.23.2.45, The location Paris, France

The benefits of SCC Workload Protection

SCC Workload Protection with Falco helps you monitor your Kubernetes clusters in real-time and identify, alert on suspicious activities and security breaches to maintain a secure environment.

SCC WP offers more than 700 default rules from various sources such as Syscall, Kubernetes Audit, and more. Additionally, you can create your custom rules.

Configuring IBM SCC Workload Protection for enhanced alert handling

In this blog, we will explore setting up the admission controller agent and Falco to receive and process alerts.

We need to send Kubernetes audit logs to security tools. Do this by transmitting the events through a Sysdig Admission Controller webhook. 

  • Admission Controller webhook:  Send the events to an external HTTP API Server (SCC Workload Protection Backend).

Deployment

You should already have the agent installed in the cluster, or if not, install them together. You can find the installation steps on the provided link: 

https://cloud.ibm.com/docs/workload-protection?topic=workload-protection-agent-deploy-kube-helm

To enable Kubernetes audit logging in your existing Sysdig Secure Helm install command, add the following flags

helm install sysdig-agent sysdig/sysdig-deploy --namespace ibm-observe --create-namespace 

--set global.sysdig.accessKey=<SERVICE_ACCESS_KEY> 

--set agent.collectorSettings.collectorHost=<INGESTION_ENDPOINT> 

--set nodeAnalyzer.nodeAnalyzer.apiEndpoint=<API_ENDPOINT> 

--set nodeAnalyzer.nodeAnalyzer.runtimeScanner.settings.eveEnabled=true 

--set nodeAnalyzer.secure.vulnerabilityManagement.newEngineOnly=true 

--set global.kspm.deploy=true 

--set nodeAnalyzer.nodeAnalyzer.benchmarkRunner.deploy=false 

--set global.clusterConfig.name=<CLUSTER_NAME> 

--set kspmCollector.apiEndpoint=<API_ENDPOINT> 

--set agent.image.registry=icr.io 

--set agent.slim.image.repository=ext/sysdig/agent-slim 

--set agent.slim.kmoduleImage.repository=ext/sysdig/agent-kmodule

--set admissionController.enabled=true \

--set admissionController.sysdig.secureAPIToken=<API_TOKEN> \    

--set admissionController.features.k8sAuditDetections=true \

--set admissionController.scanner.enabled=false 

 

Verify the Installation 

 

  • Install the Admission Controller on your Kubernetes cluster.

 

 

  • Check that Kubernetes policies are active. They are in IBM Workload protection > Policies > Runtime Policies:




An example scenario: Triggering the “Namespace Deleted” policy 

 

In this scenario, a user runs “kubectl delete namespace demo”. 

 

  • Trigger the following command to force unwanted audit detection:

        kubectl delete namespace demo

    • The command is sent to the Kubernetes API server. 

    • The server receives the API call and triggers the corresponding action. 

    • At the same time, The system generates a detailed record of the activities. It is known as an activity log. The log is stored to document the event. 

 

This activity log is automatically forwarded to the admission controller.

  • You can check the logs at the pod level:

      kubectl logs sysdig-agent-admissioncontroller-webhook -n ibm-observe

The agent analysis against Sysdig’s security policies. These policies are set to assess compliance, security, and other specific criteria, it compares the received logs with these policies, enabling a thorough evaluation of the operation. 

  • You should see something like this:

{"level":"info","component":"console-notifier","time":"2023-12-14T12:56:49Z","message":"K8s Namespace Deleted (user=IAM#a.fe@sysdig.com namespace=demo resp=200 decision=allow reason=)"}

In case of deviation or non-compliance, the system triggers alerts on Workload Protection.

  • Confirm that the event is displayed in Events in the Sysdig Secure UI.

 

 

 

The Falco rule that enabled us to detect this anomaly is provided below:

rule: K8s Namespace Deleted

description: Detect any attempt to delete a namespace

condition: (kactivity and non_system_user and kdelete and namespace and response_successful)

output: K8s Namespace Deleted (user=%ka.user.name namespace=%ka.target.name resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)

source: k8s_audit

While we offer a full set of rules, you can make custom rules for your needs.

Conclusion

In conclusion, it is imperative to leverage the comprehensive insights provided by audit logs, empowering you to monitor, detect, and respond effectively to potential security threats. By integrating these logs with security tools like IBM Cloud Security and Compliance Center Workload Protection, you establish a robust defense against unauthorized actions and potential vulnerabilities.

0 comments
54 views

Permalink