IBM Cloud Kubernetes Service - Secure By Default Cluster Virtual Private Cloud (VPC) Networking, the Details

As communicated in the previous Secure By Default Cluster VPC Networking blog, we will soon be releasing "Secure By Default Cluster VPC Networking". All newly created Red Hat OpenShift on IBM Cloud version 4.15 and IBM Cloud Kubernetes Service version 1.30 clusters will contain this enhancement. This blog details how we are accomplishing this design and demonstrates how to operate in this environment.

Security Group Overview

When an IKS or ROKS cluster is provisioned in a VPC, up to 4 security groups are created for that cluster:

• The worker security group
• The master VPE gateway security group
• The shared VPE gateway security group
• The LBaaS security group

Worker Security Group

When a new IBM Cloud Kubernetes Service VPC cluster is provisioned a security group is created for all the workers, or nodes, for the given cluster. The name of the security group is kube-<cluster ID> where <cluster ID> is the ID of the cluster. If new nodes are added to the cluster after the initial cluster provision those nodes will be added to the cluster security group automatically.

In the depiction above there are 2 clusters: Cluster 1 and Cluster 2, each with 3 worker nodes. The red-dotted line around the workers represents the security group that is created for the worker nodes in each cluster.

Master VPE Gateway Security Group

When a new IBM Cloud Kubernetes Service VPC cluster is provisioned a Virtual Private Endpoint (VPE) Gateway is created in the same VPC as the cluster. The purpose of this VPE gateway is to serve as a gateway to the cluster master which is managed by IBM Cloud. The VPE gateway is assigned a single IP address in each zone in the VPC in which the cluster has workers.

To allow access to a cluster's master only from its worker nodes a security group is created for each cluster master VPE gateway. A remote rule is then created that allows ingress connectivity from the cluster worker security group to the required ports on the cluster master VPE gateway.

Shared VPE Gateway Security Group

A security group that may be created when a new IBM Cloud Kubernetes Service VPC cluster is provisioned is the shared VPE gateway security group. A remote rule is then created that allows ingress connectivity from the cluster worker security group for the given cluster. This security group contains all VPE gateways that are shared by all clusters in that VPC. Shared VPE gateways are created when the first cluster in a VPC is provisioned. Today, shared VPE gateways exist for the IBM Cloud Container Registry, IBM Cloud Containers API, IBM Cloud Object Storage, and the IBM Cloud VPC API. In the future other shared VPE gateways that clusters need will also be created.

If this Shared VPE gateway security group already exists when a new IBM Cloud Kubernetes Service VPC cluster is provisioned, it will be recognized by the provisioning process and another will not be created. A remote rule will still be added between the existing Shared VPE gateway security group and the new worker security group. This is done so that connectivity is allowed from all clusters in the given VPC.

LBaaS Security Group

Finally, one last security group is created when a new IBM Cloud Kubernetes Service VPC cluster is provisioned. This is the Load Balancer as a Service (LBaaS) security group. This security group contains all LBaaS load balancers that are created for a given cluster in the VPC.

An LBaaS security group is created for each cluster and rules are added allowing only egress traffic from the load balancer to the cluster worker security group. In addition, remote rules are added to the cluster worker security group to allow ingress for incoming traffic from the load balancer. This is automated by the IBM Cloud Kubernetes Service and is described in detail in the Load Balancer Security Group Automation section below.

Load Balancer Service Security Group Automation

When a load balancer is created using a Kubernetes external load balancer service in the IBM Cloud Red Hat OpenShift Kubernetes Service or the IBM Cloud Kubernetes Service in IBM Cloud VPC, the necessary rules are added to the worker and LBaaS security groups to allow ingress traffic to the load balancer only for the specific port in which it is listening. Rules are also created on the LBaaS security group and worker security group allowing the load balancer to communicate to the backend pool member only for the nodeport(s) that the load balancer uses.

In the depiction above, Cluster 1 has been provisioned in IBM Cloud. 2 VPC application load balancers (ALBs) have been provisioned, one with IP address 1.2.3.4 and one with 1.2.3.10. An example on how to provision a load balancer service like this can be done with the following yaml:

apiVersion: v1
kind: Service
metadata:
  name: example-service
spec:
  selector:
    app: example
  ports:
    - port: 443
  type: LoadBalancer

As already stated, the LBaaS security group is created upon cluster creation. When a LoadBalancer service is created an LBaaS ALB is added to the LBaaS security group. In addition, the following rules are created in the LBaas Security Group:

• Ingress is allowed to port 443 on the ALB
• Egress to the the worker security group is allowed for the backend nodeports.
  • In the above example rules are added to allow egress to port 30010 for ALB 1 and to port 30012 for ALB 2
  • Note: This nodeport is automatically created in the cluster when a Kubernetes service of type LoadBalancer is provisioned.

The following rule is also added to the worker security group:

• Allow ingress to the nodeport ports from the LBaaS security group
  • In the above example ingress is allowed to port 30010 for ALB 1 and to port 30012 for ALB 2

Operating the Cluster

Adverse side effects can often occur to usability when implementing software security enhancements, and the usability with Secure by Default Cluster VPC Networking is no exception. We have worked relentlessly to provide the best experience with Red Hat OpenShift on IBM Cloud and IBM Cloud Kubernetes Service but there are a few impacts in which cluster operators must be aware when managing a cluster with Secure by Default Cluster VPC Networking. The following describes these impacts and how to overcome them.

Allowing Egress

One of the foreseeable issues is the usability of having public egress traffic blocked by default. API, CLI, Terraform, and UI options are available to allow all public egress traffic for users that wish to open all egress traffic flows. Some potential use cases that may have traffic blocked by default are:

• Pods that pull images from external public registries like quay.io or Docker Hub
• Pods that connect to IBM services over the public network, such as IAM or COS
• Pods that connect to other public services like Pager Duty or third-party logging/monitoring services

For cases like these, a CLI flag on ibmcloud ks cluster create: --disable-outbound-traffic-protection can be specified that creates a rule in the cluster worker security group that allows all egress traffic to 0.0.0.0/0. A checkbox in the UI will be present as well which will create the same security group rule.

Cleanup When Deleting a Cluster

When a cluster with the Secure by Default VPC Networking feature is deleted, the LBaaS security group, master VPE gateway security group, and worker security group will be removed automatically. The shared VPE gateway security group will not be removed as it may be in use by other clusters in the VPC. However, the remote rule between the shared VPE gateway security group and the deleted cluster worker security group is removed. The shared VPE gateway security group will also not be deleted, even when the last cluster in the VPC is deleted.

Future Work

The following will be available in future releases of SBD Networking:

• Removing the allow egress to 166.8.0.0/14 rule from the cluster worker security group
  • This rule is currently needed so the cluster workers can connect to certain IBM services for which we do not yet create a shared VPE gateway
  • If the cluster does not require access to any other IBM services, such as storage, this allow egress to 166.8.0.0/14 rule can be manually deleted from the worker security group, and if necessary more specific rules can be added to allow access only to specific IBM service CSEs
• Update existing clusters to SBD Networking
  • Currently, only newly provisioned IBM Cloud Kubernetes Service clusters at version 1.30+ and IBM Cloud Red Hat Openshift Kubernetes service clusters at version 4.15+ will have the SBD Networking feature enabled.
  • A future update will include an API and CLI to migrate existing clusters to SBD Networking.
• There are currently 2 APIs/CLIs that support resetting and syncing the current cluster worker and master VPE gateway security groups. These will not support the LBaaS and shared VPE gateway security groups until a future release. The documented CLIs are:
  • ibmcloud ks security-group reset
  • ibmcloud ks security-group sync

Thank You!

Thank you for reading this blog! Our team is diligently putting together documentation that covers several use cases which describes how to configure your cluster networking for several scenarios we feel customers may encounter, so please sure to check out the IBM Cloud documentation for more detail.