Introduction

Fusion Recipe offers variety of hooks that allow customization to meet specific application needs, ensuring smooth and efficient backup and recovery operations. In this article, we’ll explore a newly introduced hook in version 2.10 - Label and Annotation Hook, designed to enhance metadata management and streamline resource handling.

Background

In the Kubernetes (K8s) ecosystem, labels and annotations serve distinct purposes. Labels are primarily used for resource selection and grouping, while annotations provide metadata for informational purposes.

But scenarios like below present challenges that require a more flexible approach to metadata management.

Scenarios:

1.  Lack of label on resources: Mostly application resources on container platforms are labeled, but few are often left unlabeled. This inconsistency complicates the process of grouping resources for backup or recovery purposes. Creating separate groups for unlabeled resources, especially in the context of Velero (used by Fusion) is not an effective solution. It can increase API load and may lead to throttling by the Kubernetes API server during frequent backup operations.

2.  Lack of Native Name-Based Backup Support in Velero: Velero does not natively support backing up or restoring individual resources by name. Instead, it relies on broader filters such as namespaces, resource types, or labels. This limitation makes it difficult to back up specific resources unless workarounds like moving resources to a temporary namespace are employed.

3.  Absence of utilities to access Kubernetes APIs on application Pods: To keep container images lightweight, developers often exclude utilities like curl, kubectl, or oc. As a result, application pods cannot be consistently used to interact with the Kubernetes API for adding metadata to resources.

To address these challenges, Fusion Recipe introduces Label and Annotation Hooks. These hooks enable the dynamic addition of metadata to resources. For example, the Label Hook can be used to label specific resources for processing, thereby optimizing the creation of backup and recovery groups. This makes Fusion Recipe sequences more adaptable, efficient, and scalable.

Without wasting much time, lets jump into Label and Annotation Hooks.

Label Hook

Modifies metadata of a resource by applying new label or removing existing ones.

Sample example of Label hook:

Meaning of various fields:

1. name: Name of the label hook
2. labels: comma separated labels in key=value fashion
3. onError: fail or continue, by default set to fail. Meaning if label operation fail, hook will be marked failed
4. overwrite: Boolean true or false. If set to true, old value with existing keys will be overwritten
5. timeout: max duration for label operation completion
6. inverseOp: it could be any other label hook which can undo the label operation, in event of recipe failures. In this case, its – Label hook “filebrowser-resources-remove-labels”

Annotation Hook

Modifies metadata of a resource by applying new annotation or removing existing ones.

Let’s assume we want to preserve certain piece of information on a resource, in such situation, we can make use of Annotation Hooks. For instance, if you wish to preserve cluster version on a resource.

Sample example of Annotation hook:

  hooks:
    - labelSelector: app=fb
      name: add-annotations
      namespace: workload
      onError: fail
      timeout: 60
      ops:
        - annotations: for-backup=true,myapp.io/app-name=filebrowser-workload
          name: filebrowser-resources-add-annotations
          onError: fail
          overwrite: true
          timeout: 60
          inverseOp: filebrowser-resources-remove-annotations
        - annotations: for-backup-,myapp.io/app-name-
          name: filebrowser-resources-remove-annotations
          onError: fail
          overwrite: true
          timeout: 60
      selectResource: pod
      type: annotation

Annotation hooks share the similar fields as Label hooks; the only difference is that they use the annotations field instead of labels.

Important notes on Label and Annotation Hook

1. It is well known that modifying labels or annotations on certain cluster-scoped resources can impact overall cluster behavior. Therefore, any changes to the labels or annotations of these resources are generally discouraged and should be approached with caution, for example –

   ingressclass.kubernetes.io/is-default-class                 (Type: Annotation)
   storageclass.kubernetes.io/is-default-class                 (Type: Annotation)
   snapshot.storage.kubernetes.io/allow-volume-mode-change     (Type: Annotation)

2. On some resources user may hit permission issue (Reason: Forbidden). In these circumstances, Fusion recipe users are advised to grant permissions after inspection. For example -

   Reason: Forbidden
   HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"clusters.postgresql.k8s.enterprisedb.io is forbidden: User \\"system:serviceaccount:ibm-backup-restore:transaction-manager\\" cannot list resource \\"clusters\\" in API group \\"postgresql.k8s.enterprisedb.io\\" in the namespace \\"edb\\"","reason":"Forbidden","details":{"group":"postgresql.k8s.enterprisedb.io","kind":"clusters"},"code":403}\n' 
   
   

   These can be addressed by adding necessary permissions at namespace level or cluster level.

   (i)   Namespace level: create role and rolebindings

   kind: Role
   metadata:
     namespace: <Application namespace>
     name: transaction-manager-ibm-backup-restore-additional-permissions
   rules:
   - apiGroups: ["postgresql.k8s.enterprisedb.io"]
     resources: ["clusters"]
     verbs: ["patch", "get", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: transaction-manager-ibm-backup-restore-additional-permissions
     namespace: <Application namespace>
   subjects:
   - kind: ServiceAccount
     name: transaction-manager
     namespace: ibm-backup-restore
   roleRef:
     kind: Role
     name: transaction-manager-ibm-backup-restore-additional-permissions
     apiGroup: rbac.authorization.k8s.io

         (ii) Cluster level: create clusterrole and clusterrolebindings

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: transaction-manager-ibm-backup-restore-additional-permissions
   rules:
     - verbs:
         - get
         - list
         - patch
       apiGroups:
         - <Custom object apiGroup>
       resources:
         - <Custom object name>
   ---
   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: transaction-manager-ibm-backup-restore-additional-permissions
   subjects:
     - kind: ServiceAccount
       name: transaction-manager
       namespace: ibm-backup-restore
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: transaction-manager-ibm-backup-restore-additional-permissions
   
   

3. It works with both type of resources i.e. namespace scope and cluster scope. In case of cluster scope resources, namespace field is not required.
4. For selectResource field, old singular names for basic resource types are preserved for backward compatibility. Otherwise, resource "plural" is used for all resources. For Kubernetes corev1 resources, only resource plural is sufficient as group field is empty (or not present). And for other API resources fully qualified format is used with syntax: [<group>/<version>/]<plural>
   For example:

   pod
   pods
   deployment
   statefulset
   serviceaccounts
   apps/v1/deployments
   apps/v1/statefulsets
   db2u.databases.ibm.com/v1/db2uclusters
   rbac.authorization.k8s.io/v1/rolebindings
   

5. To specify fully qualified resource in Label/Annotation hook, say for EDB Postgres resource we want to add annotation k8s.enterprisedb.io/addons=["velero"], then Annotation Hook will be:

       - name: apply-annotation
         nameSelector: cluster-sample
         namespace: edb
         onError: fail
         ops:
           - annotations: 'k8s.enterprisedb.io/addons=["velero"]'
             name: edb-cluster-resource
             onError: fail
             overwrite: true
         selectResource: postgresql.k8s.enterprisedb.io/v1/clusters
         type: annotation
   

Summary

In this article, we explored the Label and Annotation Hooks in the IBM Fusion Backup and Restore Recipe, and how they enable the metadata modification of resources by adding or removing labels and annotations. These hooks play a key role in efficiently managing and organizing resources for backup and recovery operations.

In the next article, will dive into how to define a “condition” in check hooks and examine the nuances involved in its usage.

Acknowledgements: @Jim Smith @Chris Tan

References

https://ibmdocs-test.dcs.ibm.com/docs/en/fusion-software/2.10.0?topic=recipe-additional-hooks

https://www.ibm.com/docs/en/fusion-software/2.10.0?topic=workflows-creating-recipe

https://github.com/IBM/storage-fusion/tree/master/backup-restore/recipes