The friendship between DataPower and version control systems has always been a bit awkward. There are no clear instructions or best practices on how to integrate version control as part of DataPower development and deployment processes. The closest match is the article about DataPower application promotion strategies called “Managing WebSphere DataPower Device configurations for high availability, consistency, and control, Part 2: Application promotion strategies” written by John Rasmussen, Jim Brennan, and Tamika Moody in 2009. The article concentrates on application lifecycle management through software configuration management system using DataPower export/import functionality and it has been my guideline for many years. Whenever there has been a need for automated DataPower deployment solution I have taken the concept presented in the article and modified it based on the requirements of a customer. Even though the article provides a systematic approach to deployment it doesn’t provide any insight on version control which is understandable. A wide variety of implementations, and I’m now talking about the combination of tools and processes, makes it very difficult to recommend any specific approach on the subject. As said, my goal is not to tell how you should handle your versioning rather than give you an idea of what is possible and what we have been doing for the past year or so.
I have to say that the introduction of Docker form factor has been the most exciting thing that has happened in DataPower development lately. When DataPower is run in a container, local and config folder files are accessible from outside of DataPower runtime. This, and the fact that configurations are stored in a plain text file, gives you the possibility to manage everything using basic version control tools and thus seamlessly integrate version control mechanisms into your development and deployment processes. For example, we do most of the development locally and commit changes to local Git repositories. We use both Feature and Git Flow branching strategies based on the type of the repository. New functionalities are developed in separate feature branches and merged to integration branches through pull requests. Pull requests give us a convenient way to introduce review and collaboration mechanisms into the development process. So pretty much your standard everyday source code management activities.
Without going too much into details, I’ll try to give you an overview on how development and deployment are tied together a picture presenting the version control structure versus DataPower container structure is shown below.
As you can see from left side of the picture Git repository hierarchy is organized into two layers. On the higher level we have repositories that contain container specific code and files like for example auto-startup.cfg (default domain configuration) and auto-user.cfg (admin user definition). These repositories are used as source for build orchestration that creates the actual DataPower containers. On the lower level of the hierarchy we have repositories that contain the code of integration services, one integration per repository. The source code is structured in a similar folder structure than a DataPower file system. These repositories are connected to container repositories using Git submodule references. This gives us an easy way to manage the services running in specific containers. If we want to add a new integration service or introduce changes, we add or update submodule references. And if we want to remove something, we’ll just remove the reference.
The idea behind the Git hierarchy structure is quite simple. How do we then combine the content of a container repository and its submodules so that we have one unified folder structure like the one presented on the right side of the picture containing all the necessary files for a DataPower container? When a new build job is started, a container repository is checked out recursively from Git. Recursive checkout ensures that all the references are also fetched from version control system. Now that we have all submodules and container specific files, we create a complete file and folder structure using simple shell script. The configuration file that contains service1 related items is copied under one folder structure as config/app-domain1/service1.cfg, service2 configuration as config/app-domain1/service2.cfg, etc. Items from local folders in service repositories are copied under local/app-domain1.
Once the folder structure is complete, the next task is to ensure that objects defined in different configuration files are loaded into DataPower runtime ones the container starts. The answer is to use a “feature” called Include Configuration as presented in the next picture.