DataPower

DataPower horizontal pod autoscaling with MicroK8s

By Pedro Busko posted 10 days ago

  

The purpose of this post is to describe the steps of a lightweight Autoscaling DataPower Pods scenario, running in a laptop for example, the instructions are not Production ready recommendations and are solely for demo and POT purposes.



The Kubernetes implementation used was MicroK8s.

Hardware used

Apple Mac Mini (Late 2012)
Processor: 2.6Ghz Quad-Core Intel Core i7
Memory: 16GB RAM


Install MicroK8s

As a first step, let’s install MicroK8s on your machine by running the following commands:

Install Homebrew

Because we are doing this on a Mac, first thing we need is to install Homebrew

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"​

Install MicroK8s

Now let's install MicroK8s:

brew install ubuntu/microk8s/microk8s

microk8s install

microk8s status --wait-ready​


Troubleshooting


There are scenarios where the Microk8s installation command fails, this is because for some reason, in some cases macOS’s built-in DNS server is broken.

If that happens, we need some additional steps to finish MicroK8s installation. First let’s configure DNS inside the VM instance to use an external working DNS server.

Access the VM:

multipass shell microk8s-vm


Then let’s append the DNS server to resolv.conf, we will use 1.1.1.1 which is a free DNS service provided by CloudFlare. From inside the VM:

sudo vi /etc/resolv.conf


Append the following line:

nameserver 1.1.1.1

ATTENTION - resolv.conf file will be rewritten at logon time. You need to do this workaround every time you logon or keep the session open, an alternative is to use a custom cloud-init to set /etc/resolv.conf for you on first boot

Next step, is to install MicroK8s directly inside the VM:

sudo snap install microk8s --classic --channel=latest/stable
sudo iptables -P FORWARD ACCEPT

sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube


Now on the host machine from a new terminal window, check MicroK8s status:

microk8s status --wait-ready

Enable MicroK8s add-ons


MicroK8s is a lightweight Kubernetes implementation, and only install the bare minimum components, to have the additional capabilities installed, we need to enable the add-ons. Run the following commands to enable add-ons:

microk8s enable dashboard 
microk8s enable dns
microk8s enable ha-cluster
microk8s enable helm3
microk8s enable ingress
microk8s enable metrics-server
microk8s enable registry
microk8s enable storage


Changing Multipass instance memory and CPU

Since we are running this on a Mac, MicroK8s requires Multipass, a VM system for running
Ubuntu and other packages required by MicroK8s. The default VM created on the installation is not enough and we need to change its memory and CPU. Here we used 4 CPUs and 12GB of RAM

# stop multipassd
sudo launchctl unload /Library/LaunchDaemons/com.canonical.multipassd.plist

# edit the values
sudo vi /var/root/Library/Application\ Support/multipassd/multipassd-vm-instances.json​


{
"primary": {
"deleted": false,
"disk_space": "5368709120",
"mac_addr": "52:54:00:5c:f4:14",
"mem_size": "12884901888",
"metadata": {
},
"mounts": [
],
"num_cores": 4,
"ssh_username": "ubuntu",
"state": 0
}
}​

# start multipassd again
sudo launchctl load /Library/LaunchDaemons/com.canonical.multipassd.plist​

Install Helm


Because we are using a lightweight Kubernetes implementation, we will use Helm to install the DataPower operator that will be used later. 

brew install helm​


Install DataPower operator


Here are the steps to install DataPower operator:

First you download 1.3.1 release.

Unpack it

cd charts/stable/datapower-operator

helm install datapower-operator .

Create DataPowerService operand


For testing purpose, we will be using an XML Firewall that is simply a passthrough to a public API, on this case hosted on mockable.io, the XML FW is configured on the xmlfirewall.cfg file.

xmlfirewall.cfg
%if% available "xmlfirewall"

xmlfirewall "XMLFirewall"
  local-address "0.0.0.0" "2048"
  summary "an example XML Firewall Service"
  priority normal
  default-param-namespace "http://www.datapower.com/param/config"
  query-param-namespace "http://www.datapower.com/param/query"
  no force-policy-exec 
  monitor-processing-policy terminate-at-first-throttle
  debugger-type internal
  debug-history 25
  remote-address "demo8484097.mockable.io" "80"
  xml-manager default
  stylesheet-policy default
  max-message-size 0
  request-type json
  response-type unprocessed
  request-attachments strip
  response-attachments strip
  root-part-not-first-action process-in-order
  front-attachment-format dynamic
  back-attachment-format dynamic
  mime-headers 
  rewrite-errors
  delay-errors 
  delay-errors-duration 1000
  soap-schema-url "store:///schemas/soap-envelope.xsd"
  wsdl-response-policy off
  no firewall-parser-limits 
  bytes-scanned 4194304
  element-depth 512
  attribute-count 128
  max-node-size 33554432
  forbid-external-references 
  max-prefixes 1024
  max-namespaces 1024
  max-local-names 60000
  attachment-byte-count 2000000000
  attachment-package-byte-count 0
  external-references forbid
  credential-charset protocol
  ssl-config-type proxy
exit

http "XMLFirewall" xmlfirewall
  http-timeout 120
  persistent-timeout 180
  host-rewriting "on"
  no silence-warning 
  no compression 
  no include-response-type-encoding
  no always-show-errors
  no disallow-get
  no disallow-empty-reply
  persistent-connections 
  client-address "X-Client-IP"
  http-global-tranID-label "X-Global-Transaction-ID"
  httpproxy-port 800
  version "HTTP/1.1" "HTTP/1.1"
  no chunked-uploads 
exit

%endif%​


And for the DataPower operand custom resource, a YAML file (my_datapowerservice_cr.yaml) is used. This is where the DataPower CPU, Memory, and Autoscaling is configured.

For testing purposes, we are setting up the DataPower resources with the bare minimal supported, 1 initial replica with 1 CPU and 4Gi RAM. And Horizontal Pod Autoscaling with a 10% target CPU utilization and max of 2 replicas.

my_datapowerservice_cr.yaml
apiVersion: datapower.ibm.com/v1beta3
kind: DataPowerService
metadata:
  name: mydatapower
spec:
  license:
    accept: true
    use: developers-limited
    license: L-RJON-BYDQZF
  resources:
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      memory: 4Gi
  podAutoScaling:
    method: HPA
    hpa:
      targetCPUUtilizationPercentage: 10
      targetMemoryUtilizationPercentage: 0
      maxReplicas: 2
      minReplicas: 1
  domains:
  - name: "test-domain"
    dpApp:
      config:
      - "xmlfirewall-config"
  version: 10.0-cd
  replicas: 1
  users:
  - name: admin
    passwordSecret: admin-credentials
    accessLevel: privileged​

These are the commands to create all the objects necessary for the DataPower operand:

microk8s kubectl create secret generic admin-credentials --from-literal=password=admin

microk8s kubectl create configmap xmlfirewall-config --from-file=xmlfirewall.cfg

microk8s kubectl apply -f my_datapowerservice_cr.yaml​

Create Service


There are multiple ways to expose a service to external network requests, the simplest way is using NodePort. NodePort is a configuration setting you declare in a service’s YAML. Set the service spec’s type to NodePort. Then, Kubernetes will allocate a specific port on each Node to that service, and any request to your cluster on that port gets forwarded to the service.



On this case we created a service called xmlfirewall, defined on xmlfirewall_service.yml. Port 2048 is the one used by the XML Firewall.

xmlfirewall_service.yml
apiVersion: v1
kind: Service
metadata:
  name: xmlfirewall
spec:
  type: NodePort
  selector:
    app.kubernetes.io/component: datapower
    app.kubernetes.io/instance: default-mydatapower
  ports:
  - protocol: TCP
    port: 2048
    targetPort: 2048
    name: xmlfirewall​

These are the commands to create the Service that will be exposed using NodePort:

microk8s kubectl apply -f xmlfirewall_service.yml

microk8s kubectl get service​

You should write down the node port, you will need later.

Next step is to get the Multipass VM IP address that can be accessed from the host machine, your laptop for example.

multipass list

Write down the IP address that belongs to the same network as the host machine.

The service endpoint will be http://<<ip address>>:<<node port>>

Load Test

For the Load Test, we will use SOAP UI and the LoadTEST feature. Here are the steps:

Download and Install SOAP UI

Create new REST Request



Generate new Test Suite with a Load Test





Edit the Load Test for the desired settings



Run the Test Suite and observe the pod scaling on the Dashboard or on CLI

CLI

microk8s kubectl top pods

Scale up


Scale down


Dashboard

Scale up



Scale down



I'd like to thank @Bruno R Neves and@Aidan Harbison for the help troubleshooting and reviewing this material.

FAQ

Troubleshooting “Error: Kubernetes cluster unreachable”

mkdir /etc/microk8s
microk8s.config > /etc/microk8s/microk8s.conf
export KUBECONFIG=/etc/microk8s/microk8s.conf​

Error: .microk8s/config: no such file or directory

To fix this error, do the following steps:
cd $HOME
mkdir .microk8s
cd .microk8s
microk8s config > config​

How to access the Multipass VM?

multipass shell microk8s-vm​

How to access Kubernetes dashboard?

Simplest way is with the following command:

microk8s dashboard-proxy​

#write down the token


https://127.0.0.1:10443

If that fails, there is a second alternative:

token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token​

#write down the token

microk8s kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443​

https://127.0.0.1:10443

How to access DataPower WebGUI?

microk8s kubectl attach mydatapower-0 --stdin --tty

idg# co
idg(config)# web-mgmt
idg(config web-mgmt)# admin-state enabled
idg(config web-mgmt)# exit


microk8s kubectl port-forward mydatapower-0 9090:9090​


https://127.0.0.1:9090
admin/admin

How to start and stop MicroK8s?


microk8s start

microK8s stop​


References


https://brew.sh/

https://microk8s.io/

https://multipass.run/

https://helm.sh

https://kubernetes.io/

https://github.com/IBM/datapower-operator-chart

https://ibm.github.io/datapower-operator-doc/

https://www.soapui.org/​​​​
1 comment
108 views

Permalink

Comments

4 days ago

Thank you @Pedro Busko for an easy to follow, step by step guide!