Introduction
In our previous blog using hooks, we have seen how to run external commands, scale deployments up and down, or wait for certain conditions (such as pods starting up). In this article, we will look at a special case of exec hook thattry to run Kubernetes or OpenShift commands during data protection workflows. This could be required for getting resource information, patching resources, or deleting them.
Background
During the data protection workflows, there are often requirements to retrieve, update, or delete resource information. The most common approach to accomplish these tasks is by issuing oc or kubectl commands. However, in many cases, these binaries are not available within application pods posing challenges in executing necessary actions on K8 or OpenShift cluster.
For this task, creating a kubectl or oc helper pod can solve the purpose, but what about utilizing the existing application pods? Yes, we can use them to issue corresponding API requests. How easy is it to get these API endpoints? Let’s explore in this article.
The Kubernetes API Reference
Navigate the Kubernetes API Reference page to obtain details on any Kubernetes resource endpoint. You can select a specific version of this documentation to access version-specific information. However, a challenge remains: how can one retrieve details on custom resources of an application? Is there an easier method available? This issue will be addressed in the following section.
Get the API equivalents for oc commands
For this, we need to issue oc commands with increased log level verbosity option, say –v=8. Which will give you detailed output consisting of API endpoints, request or response headers, request or response body, status, and any other intermediate results.
For example -
1) (1) Put the db2uinstance in maintenance mode by applying annotation
oc command for this operation is -
$ oc annotate db2uinstance db2wh-example db2u.databases.ibm.com/maintenance-pause-reconcile=true
Let’s issue this command with option –v=8.
$ oc annotate db2uinstance db2wh-example db2u.databases.ibm.com/maintenance-pause-reconcile=true --v=8
I0319 03:39:24.836114 169853 loader.go:372] Config loaded from file: /root/.kube/config
I0319 03:39:24.860211 169853 round_trippers.go:463] GET https://<HOSTNAME>:6443/apis/db2u.databases.ibm.com/v1/namespaces/db2/db2uinstances/db2wh-example
I0319 03:39:24.860262 169853 round_trippers.go:469] Request Headers:
I0319 03:39:24.860283 169853 round_trippers.go:473] Accept: application/json
I0319 03:39:24.860299 169853 round_trippers.go:473] User-Agent: oc/4.11.0 (linux/amd64) kubernetes/1928ac4
I0319 03:39:24.860314 169853 round_trippers.go:473] Authorization: Bearer <masked>
I0319 03:39:25.125779 169853 round_trippers.go:574] Response Status: 200 OK in 265 milliseconds
I0319 03:39:25.125851 169853 round_trippers.go:577] Response Headers:
...
I0319 03:39:25.126225 169853 request.go:1073] Response Body: {"apiVersion":"db2u.databases.ibm.com/v1","kind":"Db2uInstance","metadata":{"annotations":{"db2u/certs-api-cert":"[secure]","db2u/certs-api-key":"[secure]","db2u/certs-wv-rest":"[secure]","db2u/license":"[secure]","db2u/sshkeys-db2instusr":"[secure]","db2u/sshkeys-db2uadm":"[secure]","db2u/sshkeys-db2uhausr":"[secure]","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"db2u.databases.ibm.com/v1\",\"kind\":\"Db2uInstance\",\"metadata\":{\"annotations\":{},\"name\":\"db2wh-example\",\"namespace\":\"db2\"},\"spec\":{\"account\":{\"privileged\":true},\"environment\":{\"authentication\":{\"ldap\":{\"enabled\":false}},\"databases\":[{\"dbConfig\":{\"APPLHEAPSZ\":\"25600\",\"LOGPRIMARY\":\"50\",\"LOGSECOND\":\"35\",\"STMTHEAP\":\"51200 AUTOMATIC\"},\"name\":\"BLUDB\"}],\"dbType\":\"db2wh\",\"instance\":{\"dbmConfig\":{\"DIAGLEVEL\":\"2\"},\"registry\":{\"DB2_4K_DEVICE_SUPPORT\":\"NO\",\"DB2_ATS_ENABLE\":\"NO\",\"DB2_DISPATCHER_PEEKTIMEOUT\":\"2\",\"DB2_OBJECT_STORAGE_SETTINGS\":\"OFF\"}},\"partiti [truncated 4780 chars]
I0319 03:39:25.129288 169853 request.go:1073] Request Body: {"metadata":{"annotations":{"db2u.databases.ibm.com/maintenance-pause-reconcile":"true"}}}
I0319 03:39:25.129395 169853 round_trippers.go:463] PATCH https://<HOSTNAME>:6443/apis/db2u.databases.ibm.com/v1/namespaces/db2/db2uinstances/db2wh-example?fieldManager=kubectl-annotate
I0319 03:39:25.129422 169853 round_trippers.go:469] Request Headers:
I0319 03:39:25.129446 169853 round_trippers.go:473] Accept: application/json
I0319 03:39:25.129473 169853 round_trippers.go:473] Content-Type: application/merge-patch+json
I0319 03:39:25.129497 169853 round_trippers.go:473] User-Agent: oc/4.11.0 (linux/amd64) kubernetes/1928ac4
I0319 03:39:25.129524 169853 round_trippers.go:473] Authorization: Bearer <masked>
I0319 03:39:25.230297 169853 round_trippers.go:574] Response Status: 200 OK in 100 milliseconds
I0319 03:39:25.230366 169853 round_trippers.go:577] Response Headers:
...
I0319 03:39:25.230725 169853 request.go:1073] Response Body: {"apiVersion":"db2u.databases.ibm.com/v1","kind":"Db2uInstance","metadata":{"annotations":{"db2u.databases.ibm.com/maintenance-pause-reconcile":"true","db2u/certs-api-cert":"[secure]","db2u/certs-api-key":"[secure]","db2u/certs-wv-rest":"[secure]","db2u/license":"[secure]","db2u/sshkeys-db2instusr":"[secure]","db2u/sshkeys-db2uadm":"[secure]","db2u/sshkeys-db2uhausr":"[secure]","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"db2u.databases.ibm.com/v1\",\"kind\":\"Db2uInstance\",\"metadata\":{\"annotations\":{},\"name\":\"db2wh-example\",\"namespace\":\"db2\"},\"spec\":{\"account\":{\"privileged\":true},\"environment\":{\"authentication\":{\"ldap\":{\"enabled\":false}},\"databases\":[{\"dbConfig\":{\"APPLHEAPSZ\":\"25600\",\"LOGPRIMARY\":\"50\",\"LOGSECOND\":\"35\",\"STMTHEAP\":\"51200 AUTOMATIC\"},\"name\":\"BLUDB\"}],\"dbType\":\"db2wh\",\"instance\":{\"dbmConfig\":{\"DIAGLEVEL\":\"2\"},\"registry\":{\"DB2_4K_DEVICE_SUPPORT\":\"NO\",\"DB2_ATS_ENABLE\":\"NO\",\"DB2_DISPATCHER_PEEKTIMEOUT [truncated 5091 chars]
db2uinstance.db2u.databases.ibm.com/db2wh-example annotated
It gives detailed output about the API request.
Note: This oc annotation operation turns to be API PATCH request.
1) (2) Delete the Maximo core CSV to reconcile it.
oc command for this operation is -
$ oc delete csv ibm-mas.v8.11.7
when invoked with option –v=8, we get delete API request.
$ oc delete csv ibm-mas.v8.11.7 --v=8
I0319 04:40:10.887455 170589 loader.go:372] Config loaded from file: /root/.kube/config
I0319 04:40:10.917777 170589 request.go:1073] Request Body: {"propagationPolicy":"Background"}
I0319 04:40:10.917922 170589 round_trippers.go:463] DELETE https://<HOSTNAME>:31814/apis/operators.coreos.com/v1alpha1/namespaces/mas-cpst3-core/clusterserviceversions/ibm-mas.v8.11.7
I0319 04:40:10.917946 170589 round_trippers.go:469] Request Headers:
I0319 04:40:10.917972 170589 round_trippers.go:473] Accept: application/json
I0319 04:40:10.917991 170589 round_trippers.go:473] Content-Type: application/json
I0319 04:40:10.918007 170589 round_trippers.go:473] User-Agent: oc/4.11.0 (linux/amd64) kubernetes/1928ac4
I0319 04:40:10.918025 170589 round_trippers.go:473] Authorization: Bearer <masked>
I0319 04:40:11.096900 170589 round_trippers.go:574] Response Status: 200 OK in 178 milliseconds
I0319 04:40:11.096984 170589 round_trippers.go:577] Response Headers:
...
I0319 04:40:11.097164 170589 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","details":{"name":"ibm-mas.v8.11.7","group":"operators.coreos.com","kind":"clusterserviceversions","uid":"d1f749f1-7d5b-4819-a133-b2d0fa87cf85"}}
clusterserviceversion.operators.coreos.com "ibm-mas.v8.11.7" deleted
I0319 04:40:11.097716 170589 round_trippers.go:463] GET https://<HOSTNAME>:31814/apis/operators.coreos.com/v1alpha1/namespaces/mas-cpst3-core/clusterserviceversions?fieldSelector=metadata.name%3Dibm-mas.v8.11.7
I0319 04:40:11.097755 170589 round_trippers.go:469] Request Headers:
...
I0319 04:40:11.125853 170589 round_trippers.go:574] Response Status: 200 OK in 28 milliseconds
I0319 04:40:11.125911 170589 round_trippers.go:577] Response Headers:
...
I0319 04:40:11.126145 170589 request.go:1073] Response Body: {"apiVersion":"operators.coreos.com/v1alpha1","items":[],"kind":"ClusterServiceVersionList","metadata":{"continue":"","resourceVersion":"149614443"}}
d.
Framing API requests
So far, we have gathered API details for a given oc command. To utilize this, a proper API request should be framed. Please take a look at accessing the Kubernetes API from a pod, for creating any such request. A sample example of PATCH request is available in the Storage Fusion Recipe repository.
For above example (1) Put the db2uinstance in maintenance mode by applying annotation, API request will be -
$ curl -X PATCH --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer `cat /var/run/secrets/kubernetes.io/serviceaccount/token`" -H 'Content-Type: application/merge-patch+json' -H 'Accept: application/json' -k https://kubernetes.default.svc/apis/db2u.databases.ibm.com/v1/namespaces/db2/db2uinstances/db2wh-example?fieldManager=kubectl-annotate -d '{"metadata":{"annotations":{"db2u.databases.ibm.com/maintenance-pause-reconcile":"true"}}}'
Congratulations! This is the exact API request will be used for annotating db2uinstance resource, eliminating the need of oc or kubectl binary on application Pod or additional helper pod.
Conclusion
In this article, we explored ways to get API equivalents for oc or kubectl commands, which can be used in situations where these binaries are not available. Look out for our next blog post “Fusion Recipe Tips” where we explore another aspect of using Fusion Recipes for data protection.