IBM Fusion

IBM Fusion

Ask questions, exchange ideas, and learn about IBM Fusion

 View Only

How to make use of Fusion Job hooks?

By Sandeep Prajapati posted 23 days ago

  

What are hooks, in context of application protection?

IBM Fusion Backup & Restore recipes allow you to orchestrate backup and restore workflows. One of the elements of these recipes are “hooks”. In general, hooks help in streamlining the application backup and recovery procedures on container platforms. During this process, a hook can execute a command or custom script at any specific point in time, supporting the overall orchestration. Their common usage include - running pre backup and post recovery commands, flushing in-memory buffers, freezing writes etc.

Why a new hook - Job Hook needed?

This is really a good question! Fusion Recipe already have multiple hooks, like scale, exec and check. Then why do we need a job hook if we already have an exec hook? 

Running a simple shell commands or custom script with exec hook was fine, but as their size grows or command demands longer execution period, or need dedicated execution environment, they add additional challenges, such as – execution session kept idle for substantial period, lack of execution environment etc. Also, there is no in-line YAML support for any task. To overcome these challenges, we have introduced new hook called Job Hook, using which Fusion recipe user can submit a task as job, and just wait for its completion – success or failure.

The diagram below depicts the command execution behaviour using “exec” and “Job Hook” (This is new in Fusion 2.10).

Having above context, now it’s worth exploring further details into it.

An Overview - Fusion Job Hooks

Like other hooks, Job Hooks are also spec based – meaning, we can specify them as part of Fusion Backup and Restore recipe custom resource (CR) spec. There are two supported formats for Job hooks – (1) YAML string, and (2) JSON string. String format allows to have entire Kubernetes (K8s) Job spec as part of Recipe job definitions. When recipe is applied to the cluster, there happens webhook validations for both formats –

(a) Invalid YAML/JSON string used

(b) duplicate job names and 

(c) job names used but not defined as YAML/JSON string. 

Backup agent further validates these job spec after performing variable substitutions across the job spec. These all validations occur before Recipe execution starts.

We will cover these job string formats one by one with additional explanations.

1. Job Hook as YAML string

Scenario: Let’s say we want to backup CouchDB. Due to absence of DB lock capability and checkpoint command, we proceed with dump and restore approach. Which relies on couchbackup and couchrestore utilities. These utilities are not available on CouchDB pods. For this, we need to provide an execution environment (or Pod) where it can be executed. In this situation, we can add a custom Job encapsulating these utilities. Which can be executed as part of recipe sequence. For example -

JobHook can be defined as:

Meaning of various fields:
1. name: “couchdb-backup” and “couchdb-retsore” are Job hook names. These can be referred in Recipe sequence as “job-hook-utilities/ couchdb-backup” and “job-hook-utilities/ couchdb-restore”
2. onError: Possible values are fail/continue, by default set to “fail”
3. timeout: max wait time for job completion
4. inverseOp: It’s important to note that in Job Hook, “inverseOp” could be any other hook name including other Job hook name.
5. forceCreate: if set to “true”, will delete the existing Job before creating new one. By default, set to “false”
 
 
and corresponding Job definitions would be:
  jobs:
    - couchdb-backup: |
        apiVersion: batch/v1
        kind: Job
        metadata:
          name: sample-job
          namespace: couchdb
          labels:
            label-key1: value1 
          annotations:
            annotation-key1: value2      
        spec:
          containers:
          - env:
            - name: password
              valueFrom:
                secretKeyRef:
                  key: admin_password
                  name: c-example-couchdbcluster-m
            image: <your/path/to>/couch-backup-restore:latest
            imagePullPolicy: Always
            name: backup
            command:
            - "/bin/bash"
            - "-c"
            - "/usr/local/bin/couchbackup -d mydb -u http://admin:`printenv password`@c-example-couchdbcluster-m-0:5984 -o /mnt/mydb.txt"              
            volumeMounts:
            - mountPath: /mnt
              name: backup-volume
          imagePullSecrets:
          - name: couchbackuprestore-secret
          volumes:
          - name: backup-volume
            persistentVolumeClaim:
              claimName: backup-pvc
    - couchdb-restore: |
      ...

2. Job Hook as JSON string

The JSON string format offers the same functionality but uses JSON syntax. Putting it all together -

  hooks:
    - jobs:
        - forceCreate: false
          name: couchdb-backup
          onError: fail
          timeout: 60
          inverseOp: ""
        - forceCreate: false
          name: couchdb-restore     
          onError: fail
          timeout: 60
          inverseOp: ""
      name: job-hook-utilities
      namespace: couchdb
      onError: fail
      timeout: 60
      type: job
  jobs:
    - couchdb-backup: |
        {
          "apiVersion": "batch/v1",
          "kind": "Job",
          "metadata": {
            "annotations": {
              "annotation-key1": "value2"
            },
            "labels": {
              "label-key1": "value1"
            },
            "name": "sample-job"
          },
          "spec": {
            "containers": [
              {
                "command": [
                  "/bin/bash",
                  "-c",
                  "/usr/local/bin/couchbackup -d mydb -u http://admin:`printenv password`@c-example-couchdbcluster-m-0:5984 -o /mnt/mydb.txt"
                ],
                "env": [
                  {
                    "name": "password",
                    "valueFrom": {
                      "secretKeyRef": {
                        "key": "admin_password",
                        "name": "c-example-couchdbcluster-m"
                      }
                    }
                  }
                ],
                "image": "couch-backup-restore:latest",
                "imagePullPolicy": "Always",
                "name": "backup",
                "volumeMounts": [ 
                  {
                    "mountPath": "/mnt",
                    "name": "backup-volume"
                  }
                ]
              }
            ],
            "imagePullSecrets": [
              {
                "name": "couchbackuprestore-secret"
              }
            ],
            "volumes": [
              {
                "name": "backup-volume",
                "persistentVolumeClaim": {
                  "claimName": "backup-pvc"
                }
              }
            ]
          }
        }
    - couchdb-restore: |
        ...

In Recipe sequence, these Job hooks can be referred as

  workflows:
    - failOn: any-error
      name: backup
      sequence:
        - hook: job-hook-utilities/couchdb-backup
        ...
    - failOn: any-error
      name: restore
      sequence:
        - hook: job-hook-utilities/couchdb-restore
        ...

Benefits

  1. By using Job hook, we can run any external operation at any point in time in the Fusion Backup and Restore recipe sequence. This not only allows external command execution but is also resource efficient.
  2. All capabilities of K8s Jobs can be utilized like – ttlSecondsAfterFinished, backoffLimit, etc. Giving more control on carrying out a task.
  3. In either case, Pass or Fail, Job pod logs are collected and can be inspected later.
  4. For short-lived or simple command executions, the exec hook should be the preferred choice. For long running or complex commands, the Job hook is more appropriate.

Summary

Job Hooks address the limitations of the existing exec hook by enabling the execution of longer-running commands and providing a dedicated execution environment for external commands or scripts. This enhances the adaptability and usability of Fusion Backup and Restore recipes. In the subsequent article, we will explore another hook addition - Label and Annotation Hook.

Acknowledgements: @Jim Smith @Chris Tan

References

https://ibmdocs-test.dcs.ibm.com/docs/en/fusion-software/2.10.0?topic=recipe-additional-hooks

https://www.ibm.com/docs/en/fusion-software/2.10.0?topic=workflows-creating-recipe

https://github.com/IBM/storage-fusion/tree/master/backup-restore/recipes

0 comments
20 views

Permalink