Introduction
In the dynamic landscape of container platforms, where multiple applications seamlessly coexist, gaining a thorough understanding of the resources associated with a specific application is crucial. This knowledge proves invaluable in various scenarios, including obtaining a holistic view of all resources tied to a particular application for effective management, understanding the intricate relationships between different resources to streamline both management and troubleshooting efforts, recognizing all constituting application resources for implementing robust security measures, and knowing every resource involved for a smooth and thorough removal process when uninstalling an application cleanly. Additionally, in the realm of backup and restore operations, a clear awareness of all application resources is vital for a seamless recovery.
While utility commands exist to provide resource details in pieces, this article explores a few approaches tailored to identifying all resources for an application specifically on the Red Hat OpenShift container platform, along with their Pros and Cons.
Background
In Red Hat OpenShift, the foundational principles of logical isolation are harnessed through the deployment of applications within dedicated namespaces or projects. This architectural framework ensures a systematic and secure coexistence of multiple applications within the Red Hat OpenShift container platform. While the concept of namespaces allows us to obtain a comprehensive view of resources specific to a namespace, a unique challenge arises when considering application-centric resources that extend beyond the boundaries of a single namespace. Essentially, they exist at the cluster scope. Therefore, due consideration should be given to these cluster-scoped resources as part of the process of identifying all resources for a given application.
Approach to identify resources
(1) Use oc commands
- Straightforward way to enumerate all resources in a namespace is to - use
oc api-resources command.
$ oc api-resources --verbs=list --namespaced -o name | xargs -n 1 oc get -o name -n <namespace>
configmap/kube-root-ca.crt
configmap/mycluster-initconf
configmap/openshift-service-ca.crt
endpoints/mycluster
endpoints/mycluster-instances
...
This command lists only namespace scope resources. To retrieve all cluster resources, use the following command. Subsequently, you can filter the results for resources associated with a specific application as follows:
$ oc api-resources --verbs=list --namespaced=false -o name | xargs -n 1 oc get -o name | egrep "mysql-operator|mysql-sidecar|mysql.oracle.com|zalando.org"
- For some simple, image based deployments, shorthand
all "oc get all" can be tried, which gives commonly used resources in application deployments pertaining to a namespace.
$ oc get all
NAME READY STATUS RESTARTS AGE
pod/mycluster-0 2/2 Running 0 2m33s
pod/mycluster-1 2/2 Running 0 2m33s
pod/mycluster-2 2/2 Running 0 2m33s
pod/mycluster-router-b577dfd44-hb262 1/1 Running 0 61s
pod/mysql-operator-7cfd699f57-knbrw 1/1 Running 0 2m50s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mycluster ClusterIP 172.30.12.168 <none> 3306/TCP,33060/TCP,6446/TCP,6448/TCP,6447/TCP,6449/TCP 2m33s
service/mycluster-instances ClusterIP None <none> 3306/TCP,33060/TCP,33061/TCP 2m33s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mycluster-router 1/1 1 1 2m34s
deployment.apps/mysql-operator 1/1 1 1 2m51s
NAME DESIRED CURRENT READY AGE
replicaset.apps/mycluster-router-b577dfd44 1 1 1 2m34s
replicaset.apps/mysql-operator-7cfd699f57 1 1 1 2m51s
NAME READY AGE
statefulset.apps/mycluster 3/3 2m34s
This is nice! But, this does not list the custom resources of an application.
- To explore custom resources, we can try oc/kubectl tree? as quoted "plugin to explore ownership relationships between Kubernetes objects through ownersReferences on the objects".
Command syntax: oc/kubectl tree KIND NAME [flags]
$ oc tree ics mycluster
NAMESPACE NAME READY REASON AGE
mysql-operator InnoDBCluster/mycluster - 12m
mysql-operator ├─ConfigMap/mycluster-initconf - 11m
mysql-operator ├─Deployment/mycluster-router - 11m
mysql-operator │ └─ReplicaSet/mycluster-router-b577dfd44 - 11m
mysql-operator │ └─Pod/mycluster-router-b577dfd44-q92q6 True 9m53s
mysql-operator ├─PodDisruptionBudget/mycluster-pdb - 11m
mysql-operator ├─RoleBinding/mycluster-sidecar-rb - 11m
mysql-operator ├─Secret/mycluster-backup - 11m
mysql-operator ├─Secret/mycluster-privsecrets - 11m
mysql-operator ├─Secret/mycluster-router - 11m
mysql-operator ├─Service/mycluster - 11m
mysql-operator │ └─EndpointSlice/mycluster-kb4t6 - 11m
mysql-operator ├─Service/mycluster-instances - 11m
mysql-operator │ └─EndpointSlice/mycluster-instances-dzvjw - 11m
mysql-operator ├─ServiceAccount/mycluster-sidecar-sa - 11m
mysql-operator └─StatefulSet/mycluster - 11m
mysql-operator ├─ControllerRevision/mycluster-866646b694 - 11m
mysql-operator ├─Pod/mycluster-0 True 11m
mysql-operator ├─Pod/mycluster-1 True 11m
mysql-operator └─Pod/mycluster-2 True 11m
To some extent, knowing these ownership relationship between resources is helpful, but it is not a complete solution.
Pros:
"oc api-resources" can list all resources of an namespace and cluster resources as well.
Cons:
(a) This is quite time consuming.
(b) For cluster scope resources, need to issue separate command with modified options.
(c) Filtering on cluster resources could be challenging. For this, other application resources need to be explored.
In nutshell, using oc api-resources we can retrieve all application resources.
(2) Use kube apiserver audit logs "/var/log/kube-apiserver/audit.log"
Kube apiserver audit logs are quite informative and can be looked either by connecting to kube-apiserver Pod or by looking at its hosted node (typically its master node where the Kubernetes control plane components runs). Sample example of audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"e0431852-74f0-45c6-a970-f9fe26c73d51","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/clusterroles?fieldManager=kubectl-client-side-apply\u0026fieldValidation=Ignore","verb":"create","user":{"username":"kube:admin","groups":["system:cluster-admins","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.17.101.196"],"userAgent":"oc/4.11.0 (linux/amd64) kubernetes/1928ac4","objectRef":{"resource":"clusterroles","name":"mysql-sidecar","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-01-21T11:58:25.320637Z","stageTimestamp":"2024-01-21T11:58:25.330054Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"system:cluster-admins\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"2b749d5d-73f3-45c8-abba-992abbf47a74","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?fieldManager=kubectl-client-side-apply\u0026fieldValidation=Ignore","verb":"create","user":{"username":"kube:admin","groups":["system:cluster-admins","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.17.101.196"],"userAgent":"oc/4.11.0 (linux/amd64) kubernetes/1928ac4","objectRef":{"resource":"clusterrolebindings","name":"mysql-operator-rolebinding","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-01-21T11:58:25.465028Z","stageTimestamp":"2024-01-21T11:58:25.468828Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"system:cluster-admins\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7cc8f4c0-d71e-4507-888a-4ac3d48dfdfc","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/mysql-operator/serviceaccounts?fieldManager=kubectl-client-side-apply\u0026fieldValidation=Ignore","verb":"create","user":{"username":"kube:admin","groups":["system:cluster-admins","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.17.101.196"],"userAgent":"oc/4.11.0 (linux/amd64) kubernetes/1928ac4","objectRef":{"resource":"serviceaccounts","namespace":"mysql-operator","name":"mysql-operator-sa","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-01-21T11:58:25.954915Z","stageTimestamp":"2024-01-21T11:58:25.960360Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"system:cluster-admins\""}}
This audit.log can be properly filtered during application deployments for create events and later parsed to gather relevant information. For example:
$ tail -n 0 -f /var/log/kube-apiserver/audit.log | grep "\"verb\":\"create\"" > application-deployment-audit.log
# Parse the captured logfile
$ python parse.py application-deployment-audit.log
verb|resource|namespace|name|apiGroup|apiVersion
create|apiservices| |v2.mysql.oracle.com|apiregistration.k8s.io|v1
create|customresourcedefinitions| |mysqlbackups.mysql.oracle.com|apiextensions.k8s.io|v1
create|customresourcedefinitions| |clusterkopfpeerings.zalando.org|apiextensions.k8s.io|v1
create|apiservices| |v1.zalando.org|apiregistration.k8s.io|v1
create|customresourcedefinitions| |kopfpeerings.zalando.org|apiextensions.k8s.io|v1
create|clusterroles| |mysql-operator|rbac.authorization.k8s.io|v1
create|clusterroles| |mysql-sidecar|rbac.authorization.k8s.io|v1
create|clusterrolebindings| |mysql-operator-rolebinding|rbac.authorization.k8s.io|v1
create|clusterkopfpeerings| |mysql-operator|zalando.org|v1
create|namespaces| |mysql-operator| |v1
create|clusterserviceversions|mysql-operator|ibm-cert-manager-operator.v4.2.1|operators.coreos.com|v1alpha1
create|serviceaccounts|mysql-operator|mysql-operator-sa| |v1
create|deployments|mysql-operator|mysql-operator|apps|v1
create|serviceaccounts|mysql-operator|mysql-operator-sa| |v1
create|secrets|mysql-operator|mypwds| |v1
create|innodbclusters|mysql-operator|mycluster|mysql.oracle.com|v2
create|rolebindings|mysql-operator|mycluster-sidecar-rb|rbac.authorization.k8s.io|v1
create|statefulsets|mysql-operator|mycluster|apps|v1
create|deployments|mysql-operator|mycluster-router|apps|v1
create|serviceaccounts|mysql-operator|mycluster-sidecar-sa| |v1
...
Pros:
Gives efficient retrieval of all application resources if parsing script in place.
Cons:
This approach results could be misleading if two or more applications have been deployed together.
(3) Query etcd store records
In Kubernetes, etcd stores the configuration data, state and metadata of the entire cluster. "etcdctl" a command line utility can be used to query the etcd key-value store. A simplest way to get all resources of an application is to get all the keys and filter needed resources as follows:
$ etcdctl get / --prefix --keys-only | grep -v "events" | egrep "mysql-operator|mysql-sidecar|mysql.oracle.com|zalando.org"
/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/clusterkopfpeerings.zalando.org
/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/innodbclusters.mysql.oracle.com
/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/kopfpeerings.zalando.org
/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/mysqlbackups.mysql.oracle.com
/kubernetes.io/apiregistration.k8s.io/apiservices/v1.zalando.org
/kubernetes.io/apiregistration.k8s.io/apiservices/v2.mysql.oracle.com
/kubernetes.io/clusterrolebindings/mysql-operator-rolebinding
/kubernetes.io/clusterroles/mysql-operator
/kubernetes.io/clusterroles/mysql-sidecar
/kubernetes.io/configmaps/mysql-operator/kube-root-ca.crt
/kubernetes.io/configmaps/mysql-operator/mycluster-initconf
...
Pros:
A efficient way to get all application resources when applied proper filtering.
Cons:
Getting proper filter for cluster resources will require little effort.
Conclusion
For simple application deployments, it’s easy to find all the constituting resources. But, for any complex applications such as IBM cloud Paks, its tedious task, in such situations approaches outlined above helps in finding the application resources easily. Knowing application resources beforehand greatly helps in scenarios mentioned above and in Recipe writing. In the next blog, will discuss how to specify resources while developing Recipes for backup and restore workflows.
Additional links
Recipe documentation
Recipe public repository
Recipe updates
#Highlights
#Highlights-home