Introduction
In the previous blog, we learned how to “specify Recipe groups” to represent a set of resources in Recipe. In this article, we will explore the creation and usage of hooks in various scenarios of backup and restore workflows, emphasizing their significance.
Background
To backup container applications, only specifying application resources in Recipe groups and taking their snapshots is not always sufficient to guarantee that the application can be recovered correctly. For application consistent backups, certain commands or scripts need to be executed before and after the volume snapshot operations. Same is the case during recovery. In some cases, post actions are required after the resources are restored to recover the application. How to perform these pre and post actions in backup and restore workflows?
While backups are usually straightforward, the major challenge in creating a Recipe is to create a workflow to recover the application. During recovery, application passes through different stages of its deployment. How to validate these intermediate stages to allow smooth recovery of the application?
What are Recipe hooks?
Hooks allow us to run external commands or scripts before and after PVC snapshots (persistent data), scale deployments up and down, or wait for certain conditions (such as pods starting up). There are three hooks available, namely exec, check, and scale. For more details on these see “Specifying a hook” section in the Recipe documentation.
Using Recipe hooks
We can use hooks at various stages of the backup and recovery workflow. Let's create a few hooks to address the given scenarios.
Exec hook
It’s an important hook that allows to run commands or scripts during Recipe backup and recovery workflows. Some common cases where this can be utilized are - to flush the memory buffers, checkpoint database, databases locking, freeze a filesystem etc., so that data is backed up in a consistent state.
An example of exec hook – Flush tables before snapshotting MySQL database volumes.
hooks:
- name: mysql-pod-exec
type: exec
namespace: mysql
labelSelector: app=mysql
timeout: 60
onError: fail
ops:
- name: "flush-tables-with-read-lock"
command: >
["/bin/bash", "-c", "mysql --user=root --password=$MYSQL_ROOT_PASSWORD -e 'FLUSH TABLES WITH READ LOCK;'"]
container: mysql
In this, command “FLUSH TABLES WITH READ LOCK” is executed in the mysql container and mysql pod was selected using labelSelector app=mysql in mysql namespace.
Call this hook in backup workflow as:
workflows:
- name: backup
sequence:
...
- hook: mysql-pod-exec/flush-tables-with-read-lock
- group: mysql-volumes
...
Ability to issue flush commands before snapshot operation helps in achieving application consistency.
Check hook
There are two variants of this hook.
(1) Check hook for common Kubernetes resources such as pod, deployment and statefulsets.
An example of deployment check hook – Verify MySQL deployment ready replica count post restore is same as it was specified in the spec.
hooks:
- name: mysql-deployment-check
type: check
namespace: mysql
selectResource: deployment
nameSelector: mysql
timeout: 120
onError: fail
chks:
- name: replicasReady
timeout: 600
onError: fail
condition: "{$.spec.replicas} == {$.status.readyReplicas}"
MySQL deployment resource is selected using nameSelector in mysql namespace and validated as JSONPath expression condition.
Call this hook in restore workflow as:
- name: restore
sequence:
- group: mysql-volumes
- group: mysql-resources
- hook: mysql-deployment-check/replicasReady
Validating these conditions during recovery helps determine when to use the next operation or stop the workflow process.
(2) Check hook for custom resources.
An example of deployment check hook – Verify Db2 cluster instance is in ready state post restore.
hooks:
- name: db2ucluster-instance-check
type: check
namespace: db2
selectResource: db2u.databases.ibm.com/v1/db2uclusters
labelSelector: formation_id=db2oltp-test
timeout: 800
onError: fail
chks:
- name: stateReady
timeout: 900
onError: fail
condition: "{$.status.state} == {\"Ready\"}"
Here, du2clusters is a custom resource selected using label of fully qualified resource type <APIGROUP>/<APIVERSION>/<NAME> in the db2 namespace. Finally, validated against JSONPath expression condition.
Use this hook in restore workflow as:
- name: restore
sequence:
- group: db2u-volumes
- group: db2u-resources
...
- hook: db2ucluster-instance-check/stateReady
This check in end ensures that Db2 database is ready to use post restore.
Scale hook
Scale hooks helps during application backup by scaling application resources up or down, or can also be utilized for pod restart requirements.
An example of scale hook – Restart MongoDB pods post restore.
hooks:
- name: mongodb-cluster-scale
type: scale
namespace: ot-operators
selectResource: statefulset
labelSelector: app=mongodb-cluster,mongodb_setup=cluster,role=cluster
This is the scale hook for MongoDB statefulset selected using labelSelector in ot-operators namespace.
Use this hook in restore workflow for pod restart as:
- name: restore
sequence:
- group: mongodb-volumes
- group: mongodb-resources
...
- hook: mongodb-cluster-scale/down
- hook: mongodb-cluster-scale/sync
- hook: mongodb-cluster-scale/up
- hook: mongodb-cluster-scale/sync
Scale down brings pod count to zero, adding sync helps in meeting the desired zero count of previous operation. Similarly, scale up reverts to the original pod count, and further sync ensures all pods are available.
Summary
In this article, we explored using different hooks available in the Fusion Recipe with the help of examples. For more details on hooks refer Recipe documentation. In our next blog, will look at writing our first Recipe to get some experience with the framework.