In today’s fast-evolving technology, developers and DevOps teams are creating, releasing, and deploying applications faster than ever. Kubernetes clusters have become essential for managing these containerized applications efficiently. However, as the pace of development accelerates, ensuring the safety and security of these environments become critical. It is imperative to keep up with these rapid changes and quickly identify any anomalies.
This article explores the importance of Kubernetes audit logs, and the critical role of IBM Cloud Security Compliance Center Workload Protection in safeguarding. Additionally, we'll introduce Falco, a cloud-native security tool.
These audit logs, Falco and Workload Protection will create a strong security shield for your Kubernetes world.
Kubernetes audit logging
The Kubernetes audit logging is a detailed record of cluster activities. It captures information about user interactions, API requests, system changes, and resources access. It provides administrators a precise status and accounting of cluster actions.
What is Falco
Falco is a guardian tool that actively monitors your kubernetes cluster and detects anything suspicious in real-time, It understands security activities, generates alerts or takes action when predefined rules are violated, it analyzes log data to find abnormal behaviors and potential threats. It ensures that all suspicious activities are promptly identified and addressed.
The benefits of Kubernetes audit logs
Securing and maintaining reliable Kubernetes environments is crucial for business using containerized applications. By combining these audit logs, administrators gain the ability to address fundamental questions:
-
What happened? An attempt was made to delete a critical namespace within the kubernetes cluster.
-
Who initiated the action? The action was initiated by a user with the identity toto.tata@ibm.com
-
When did it happen? The deletion attempt occurred on March 11,2025 at 15:45 UTC
-
Where did it occur? The action originated from the IP address 10.23.2.45, The location Paris, France
The benefits of SCC Workload Protection
SCC Workload Protection with Falco helps you monitor your Kubernetes clusters in real-time and identify, alert on suspicious activities and security breaches to maintain a secure environment.
SCC WP offers more than 700 default rules from various sources such as Syscall, Kubernetes Audit, and more. Additionally, you can create your custom rules.
Configuring IBM SCC Workload Protection for enhanced alert handling
In this blog, we will explore setting up the admission controller agent and Falco to receive and process alerts.
We need to send Kubernetes audit logs to security tools. Do this by transmitting the events through a Sysdig Admission Controller webhook.
Deployment
You should already have the agent installed in the cluster, or if not, install them together. You can find the installation steps on the provided link:
https://cloud.ibm.com/docs/workload-protection?topic=workload-protection-agent-deploy-kube-helm
To enable Kubernetes audit logging in your existing Sysdig Secure Helm install command, add the following flags
helm install sysdig-agent sysdig/sysdig-deploy --namespace ibm-observe --create-namespace
--set global.sysdig.accessKey=<SERVICE_ACCESS_KEY>
--set agent.collectorSettings.collectorHost=<INGESTION_ENDPOINT>
--set nodeAnalyzer.nodeAnalyzer.apiEndpoint=<API_ENDPOINT>
--set nodeAnalyzer.nodeAnalyzer.runtimeScanner.settings.eveEnabled=true
--set nodeAnalyzer.secure.vulnerabilityManagement.newEngineOnly=true
--set global.kspm.deploy=true
--set nodeAnalyzer.nodeAnalyzer.benchmarkRunner.deploy=false
--set global.clusterConfig.name=<CLUSTER_NAME>
--set kspmCollector.apiEndpoint=<API_ENDPOINT>
--set agent.image.registry=icr.io
--set agent.slim.image.repository=ext/sysdig/agent-slim
--set agent.slim.kmoduleImage.repository=ext/sysdig/agent-kmodule
--set admissionController.enabled=true \
--set admissionController.sysdig.secureAPIToken=<API_TOKEN> \
--set admissionController.features.k8sAuditDetections=true \
--set admissionController.scanner.enabled=false
Verify the Installation
An example scenario: Triggering the “Namespace Deleted” policy
In this scenario, a user runs “kubectl delete namespace demo”.
kubectl delete namespace demo
-
-
The command is sent to the Kubernetes API server.
-
The server receives the API call and triggers the corresponding action.
-
At the same time, The system generates a detailed record of the activities. It is known as an activity log. The log is stored to document the event.
This activity log is automatically forwarded to the admission controller.
kubectl logs sysdig-agent-admissioncontroller-webhook -n ibm-observe
The agent analysis against Sysdig’s security policies. These policies are set to assess compliance, security, and other specific criteria, it compares the received logs with these policies, enabling a thorough evaluation of the operation.
{"level":"info","component":"console-notifier","time":"2023-12-14T12:56:49Z","message":"K8s Namespace Deleted (user=IAM#a.fe@sysdig.com namespace=demo resp=200 decision=allow reason=)"}
In case of deviation or non-compliance, the system triggers alerts on Workload Protection.
The Falco rule that enabled us to detect this anomaly is provided below:
rule: K8s Namespace Deleted
description: Detect any attempt to delete a namespace
condition: (kactivity and non_system_user and kdelete and namespace and response_successful)
output: K8s Namespace Deleted (user=%ka.user.name namespace=%ka.target.name resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
source: k8s_audit
While we offer a full set of rules, you can make custom rules for your needs.
Conclusion
In conclusion, it is imperative to leverage the comprehensive insights provided by audit logs, empowering you to monitor, detect, and respond effectively to potential security threats. By integrating these logs with security tools like IBM Cloud Security and Compliance Center Workload Protection, you establish a robust defense against unauthorized actions and potential vulnerabilities.