The Ugly History

While everyone says protecting a company’s data is protecting their most valuable asset, data protection has always been a second-class citizen in the IT hierarchy. Phrases like:

• Selling backup software is like selling homeowners insurance, hard to show value without showing pictures of burnt houses.
• Doing backup is like taking out the garbage, the business doesn’t notice until it isn’t done.
• Nobody gets promoted for implementing a good backup system, but they get fired when it doesn’t work.

I can remember the early days of open-systems Disaster Recovery (DR) discussions. A company would have a mission critical application deployed on big Unix iron and then decided they needed a DR solution in case something happened to the Datacenter. (Had a very interesting conversation with a customer who realized the factory they converted to a DC was in a flood plain.) Data replication was a relatively simple conversation about RPO/RTO and the speed of tlight until we pointed out that the easy part of the solution was getting the data to the target. The customer still had a lot of work to get the application up and running again. The common thread to both of these is protecting data and applications was an afterthought in the solution development process.

“Those that fail to learn from history you are doomed to repeat it.”¹

IBM has a long and storied reputation for building infrastructure to enable the highest levels of application availability and reliability. It is amazing how much critical business processing runs on Mainframes 25 years² after they were declared to be dead. IBM (with the help of Red Hat) is leveraging its pedigree to change the way we think about application availability, data protection, and disaster recovery so that it becomes a first-class citizen in the cloud-native application development process.

Red Hat OpenShift (Kubernetes) addresses part of application availability challenges by the way resource requirements are defined. A developer defines the desired state of the resources for their application. OpenShift then monitors the environment to address any gaps between current state and desired state. So, unlike the old DR days, where external processes had to be constructed to monitor “good” and manage the restart an application, cloud-native applications
have the basic knowledge encoded to get (re)started. So, the task for IBM is to leverage what Red Hat has built and add data protection and replication tooling. But wait...

Adding CP to CI/CD

To realize the full value of containers and digital transformation, an organization needs to adopt Continuous Integration/ Continuous Deployment (CI/CD). Gone are the 6 month test and QA cycles where the backup and DR infrastructure teams could test their processes to ensure they have accounted for all changes in the application structure. Now minor tweaks are coming out in hours and significant functional changes are coming out every few weeks. Even if we wanted to use the old “afterthought” approach, it would be very risky. (Remember people get fired when restores fail to return operations to the business.)

It is time to add Continuous Protection to our rapid integration and deployment processes. But how do we add backup and DR technology, that has traditionally bolted on to an infrastructure into fast moving CI/CD environments. The answer is we leverage OpenShift constructs of Custom Resource Definitions (CRD) (object definition) and Custom Resources (CR) (object instance). If you are familiar with OpenShift, these are the same constructs that developers use
to define how their applications are deployed and the expect states. (If you are not an OpenShift SME, just drop “CRD” and “CR” to your OpenShift compadres to show them you can talk their language.)

So, each application will have CRs that define how the application namespace should be backed up and how it is properly restored (if needed). Optionally, critical applications can have CRs that define DR policies and failover processes. These CRs are maintained with the application code, so if changes in the application structure affect the way the application is backed up or the application is failed over, the CRs are updated in the development process.

OpenShift Architects and Administrators to the Rescue

But you say, “My app/dev SMEs aren’t backup/DR SMEs”. How easy is it to add CP to CI/CD? This is where CRDs come into play. These are templates created and maintained by the people who know the most about your OpenShift infrastructure. Templates define how frequently backups are taken, provide hooks for pre-and post-backup processing (such as quiesce / release a database), backup retention and storage location. For more information on the details of backup CRs, please see a great blog by JC Lopez, IBM Storage Fusion Backup as Code. Similar CRDs can be created for DR, to be used by mission critical application developers. For more information the breadth or the Fusion Storage Solution for application resiliency, see Venkat Kolli’s Blog Introduction to application resiliency.

Remember the statement above: “Doing backup is like taking out the garbage, the business doesn’t notice until it isn’t done”. In Storage Fusion 2.6 coming in Q3/2023, a dashboard will be available that monitors the status of backup jobs on one or more OpenShift clusters in your environment, including automatically identifying which application namespaces are not being backed up, so the OpenShift admin will know before the business does, that an application is not being backed up.

Conclusion

Treating backup and DR as components of the CI/CD process, is new and still evolving, but you can be confident that IBM is leveraging its experience in application availability and reliability to bring the tools and technologies you need to successfully deploy and support your most mission critical applications on OpenShift.

¹ Winston Churchill - 1948
² A Quick Look Back – The (NOT) Demise of the IBM Mainframe, Keith Allingham, May 27, 2021