DataPower

The friendship between DataPower and version control systems has always been a bit awkward. There are no clear instructions or best practices on how to integrate version control as part of DataPower development and deployment processes. The closest match is the article about DataPower application promotion strategies called “Managing WebSphere DataPower Device configurations for high availability, consistency, and control, Part 2: Application promotion strategies” written by John Rasmussen, Jim Brennan, and Tamika Moody in 2009. The article concentrates on application lifecycle management through software configuration management system using DataPower export/import functionality and it has been my guideline for many years. Whenever there has been a need for automated DataPower deployment solution I have taken the concept presented in the article and modified it based on the requirements of a customer. Even though the article provides a systematic approach to deployment it doesn’t provide any insight on version control which is understandable. A wide variety of implementations, and I’m now talking about the combination of tools and processes, makes it very difficult to recommend any specific approach on the subject. As said, my goal is not to tell how you should handle your versioning rather than give you an idea of what is possible and what we have been doing for the past year or so. 

I have to say that the introduction of Docker form factor has been the most exciting thing that has happened in DataPower development lately. When DataPower is run in a container, local and config folder files are accessible from outside of DataPower runtime. This, and the fact that configurations are stored in a plain text file, gives you the possibility to manage everything using basic version control tools and thus seamlessly integrate version control mechanisms into your development and deployment processes. For example, we do most of the development locally and commit changes to local Git repositories. We use both Feature and Git Flow branching strategies based on the type of the repository. New functionalities are developed in separate feature branches and merged to integration branches through pull requests. Pull requests give us a convenient way to introduce review and collaboration mechanisms into the development process. So pretty much your standard everyday source code management activities.

Without going too much into details, I’ll try to give you an overview on how development and deployment are tied together a picture presenting the version control structure versus DataPower container structure is shown below.

As you can see from left side of the picture Git repository hierarchy is organized into two layers. On the higher level we have repositories that contain container specific code and files like for example auto-startup.cfg (default domain configuration) and auto-user.cfg (admin user definition). These repositories are used as source for build orchestration that creates the actual DataPower containers. On the lower level of the hierarchy we have repositories that contain the code of integration services, one integration per repository. The source code is structured in a similar folder structure than a DataPower file system. These repositories are connected to container repositories using Git submodule references. This gives us an easy way to manage the services running in specific containers. If we want to add a new integration service or introduce changes, we add or update submodule references. And if we want to remove something, we’ll just remove the reference.

The idea behind the Git hierarchy structure is quite simple. How do we then combine the content of a container repository and its submodules so that we have one unified folder structure like the one presented on the right side of the picture containing all the necessary files for a DataPower container? When a new build job is started, a container repository is checked out recursively from Git. Recursive checkout ensures that all the references are also fetched from version control system. Now that we have all submodules and container specific files, we create a complete file and folder structure using simple shell script. The configuration file that contains service1 related items is copied under one folder structure as config/app-domain1/service1.cfg, service2 configuration as config/app-domain1/service2.cfg, etc. Items from local folders in service repositories are copied under local/app-domain1.

Once the folder structure is complete, the next task is to ensure that objects defined in different configuration files are loaded into DataPower runtime ones the container starts. The answer is to use a “feature” called Include Configuration as presented in the next picture.

The left side of the picture shows text representations of include configurations that refer to external configuration files. The right side of the picture shows the same include configurations and what they look like in DataPower WebGUI. As you can see from the picture include configurations are human-readable and thus can be easily created automatically as part of the build orchestration.

To summarize the key ideas, dependencies between container and service objects is handled using Git submodule references. Container folder structure is constructed using automated script. Include configurations are added automatically to an application domain configuration file based on the contents of config folder. A high-level example of a flow from development into runtime looks something like the one presented in the following picture.

A developer modifies DataPower configuration using for example a local DataPower Docker instance. Changes can be made directly to version control by assigning config and local folders, that reside in a local service level Git repository, as volume parameters to docker run command. Once modifications are ready, they are committed to the local repository and then pushed into a remote repository. The service repository changes are included into a container level repository by updating the submodule references. This activity starts the build process that produces a Docker image which is then pushed into a private Docker registry. Release automation framework is responsible of the deployment tasks that move images from the Docker registry into container runtime as DataPower instances. Release automation is separated from the build orchestration in this example because in our environment they are totally different functional components.

I hope some of you find this post useful and get some new ideas on how benefit from version control systems as part of your DataPower development and deployment processes. If you have any questions, feel free to throw in a comment.