DevOps Automation

This article was originaly published in 2017.08.04

Summary

IBM DataPower is a powerful SOA tool to provide complete control over your various workloads. Through the use a central controller, one can quickly give or revoke access to critical resources via a communication network. However, when it comes to manually configuring this tool, it can be quite daunting. IBM DataPower offers many modification possibilities and a small error may cause major outages. The IBM DataPower plug-in for IBM UrbanCode Deploy offers a wide set of tools to help organize, deploy, and rollback your DataPower configurations. The DataPower Configuration Manager (DCM) project offers a fully supported, but open source, approach to help solve your toughest DataPower configuration and deployment scenarios. The DCM is an Ant-based command line tool that offers DataPower modification and configuration functionality. The DataPower plug-in is built around this DCM to provide the same powerful functionality with ease from within IBM UrbanCode Deploy. Combined with IBM UrbanCode Deploy's natural deployment capabilities, anyone can manage DataPower's complex configurations with the push-of-a-button deployments. While we offer much of the tool's functionality through our plug-in, all DCM command line commands can be also run through a simple IBM UrbanCode Deploy Shell step. In the walkthrough below, we have outlined a simple use case of importing a sample DataPower domain configuration export.zip into a DataPower environment. This is a great tutorial to follow if you have never used IBM DataPower or the DataPower plug-in before. In our next walkthrough, we will utilize a definition file to help add, remove, and configure specific objects from a sample export.zip. The DCM's power shines with its proper use of the definitions file. Then combined with IBM UrbanCode Deploy, one can easily manage DataPower artifacts and deployment processes to make updates and modifications easily repeatable. For now, we will stick to the initial deployment requirements and import basics.

Requirements

All deployments, simple or complex, will require at minimum a IBM UrbanCode Deploy server, IBM UrbanCode Deploy agent, and an IBM DataPower server. The DataPower server can be either physical or virtual.

If you already have a physical DataPower server available, the IBM UrbanCode Deploy agent can be placed on any system with network access to the DataPower server. Configuration changes are made via SOAP commands which do not require other physical scripts or libraries. The agent cannot be installed on the DataPower server. Once IBM UrbanCode Deploy and DataPower servers are running, download the IBM DataPower plug-in and install it into the IBM UrbanCode Deploy server. After the installation is complete, you will see the DataPower plug-in steps in the process designer's step palette.

Walkthrough

1. Download a domain configuration export.zip from your DataPower server

   1. Before we begin designing our IBM UrbanCode Deploy process, we first need to identify what we are going to import. In this example, we will use the current default configuration of your DataPower server. The simplest way to retrieve this export is to navigate, via browser, to your DataPower server. Once logged in, open the Menu Drawer in the top left and click Export Configuration.
      
   2.  IBM DataPower Gate Menu Drawer
      1. Follow the prompts to create a complete export.zip of your domain configuration. To ensure there are configuration objects in the export.zip, make sure you export all objects from your domain. Otherwise, you will find yourself with an empty export with nothing to configure.
      2. After clicking the Download button, your browser should begin downloading the export.zip file. This file will become the Component Version artifact in IBM UrbanCode Deploy and what we will deploy to the DataPower environment.

2. Create a new DataPower component in IBM UrbanCode Deploy and add the new component version

   1. Navigate to IBM UrbanCode Deploy in a browser and login.
   2. Click the large Component tab and then the Create Component button.
   3. Begin to fill out the component creation and set a component name and source configuration. The name of the component is irrelevant, but make sure you remember it for later. For this walkthrough series, I named my component DataPower Demo 1 Component. The second most import setting is the source configuration. This is how we will import our export.zip into IBM UrbanCode Deploy and create a new component version. Select and configure a source configuration type that you either mirrors your preconfigured source control or design a new File System (Versioned) for simplicity. If it is your first time importing Component Versions, follow the Creating components from a versioned file system documentation. Click Save to confirm your new component.
      
       IBM UrbanCode Deploy Component Creation 
      
      
   4. Next, import the DataPower artifacts. Move to the Versions sub-tab and click the Import Versions button. If successful, you should see a new component version as in the screenshot below.
      

3. Create the Import Basic component process

   1. From your new DataPower component, navigate to the Processes sub-tab, located next to the Versions sub-tab, and click the Create Processes button. Create and save a new process. In the screenshot below, you can see I have named my process Import Basic as it focuses around the Import (Basic) plug-in step.
      
      
       
   2. Below you will see the basic outline to our Basic Import DataPower deployment process. It is composed of four steps that are each integral in identifying the correct DataPower export.zip artifact and deploying it into the DataPower environment. The first two steps are common and inherent IBM UrbanCode Deploy steps, while the second two are from the DataPower plug-in. Each of these steps can be found in the step palette on the left-hand side of the process designer.
      
      
      
       DataPower Import Basic Component Process
      
      Delete Files and Directories: This step first cleans the working directory. After you run several deployments without this step, you may find that the working directory has become cluttered with old files. This step will delete the contents of the working directory, so you can be confident previously deployed files will not accidentally be used. Download Artifacts: The next step will resolve the selected component artifacts. In our demo, this will be the export.zip file we recently imported as a component version. This is a dynamic step, as the version selected at deployment time will define the artifacts to be resolved. Notice that a version is not defined within the step configuration.
      
       Deployment Preparation Steps
      
      
   3. Import (Basic): This is the first DataPower plug-in specific step and does the heavy lifting of importing the export.zip's contents into DataPower. The Import (Basic) step is the most important step in this process. Save Configuration: The final step will confirm and save the changes in IBM DataPower Gateway. If this step is not run, you will find an Unsaved Changes alert message across all pages in DataPower. Note: The Import (Basic) and Save Configuration steps make use of application, resource, and environment properties. These will be configured in the following walkthrough step. For now, configure each step as shown in the screenshot.
      
       DataPower plug-in Import (Basic) and Save Configuration step
      
      
   4. Once all steps have been configured, Save the new process.

4. Create Application, Environment, and required properties

   1. To create a new Application, select the Application tab and then the Create Application button. The Application's name for this walkthrough is DataPower Demo Application.
   2. Next create an Environment by clicking the Create Envrionment button within your new Application. It is important that you name this environment the same as the DataPower environment you plan to deploy to. If you look closely at the Datapower steps' configuration, you will see the environment property's value set as ${p:environment.name}. Based on the environment you deploy against, this property will be updated with the environment name. In my Docker DataPower instance, this environment is called dev. Setting this environment name is important if you wish to take advantage of advanced functionality in the definitions file.
      
       IBM UrbanCode Application and Environment
      
      
   3. Once the environment is created, click its name to configure it. At this point, you will want to attach a resource agent to the selected environment. Directions on the knowledge center show how to map environments to applications. As mentioned above, the agent can be located anywhere with network access to the Datapower server. The agent should not be located on the DataPower server.
   4. Now that your application environments are mapped, we will configure the necessary environment specific properties. Environment property configuration can be found by clicking the Configuration sub-tab and then the Environment Properties menu item from the left-hand side. Here, you will want to specify the same property names with your environment specific host, user, password, and port properties.
      
       DataPower Environment property configuration
      
      In the future, if you create a test or prod DataPower environment, you would create and configure the environment specific properties in similar fashion. On deployment, the component process will be able to dynamically resolve the property values based on the environment it is run against.
   5. Our last property that we still need to specify will be located under the Application property configuration. This can be found at the main DataPower Demo Application page under Configuration sub-tab and Application Properties menu item.
      
       DataPower Application property configuration.
      
      These Application properties are created identical to the Environment properties and are resolved in similarly. With this configuration, any process run under this application will have access to that domain property value of default. In the Docker DataPower environment, default is the preset domain's name. At this point, all required properties are configured and ready to be used. Note: The ${p:componentProcess.defaultWorkDir} property is a default property that is available on all components. This property is automatically set at runtime and is based on the runtime agent and component's name.

5. Component Assignment and Application Process Creation

   1. Before we can finalize our deployment process, we need to specify what components are available to this application. Simply navigate to the Components sub-tab of the Application and select Add Component. Identify the DataPower component we created previously and add it to you Application. Once added, it should look like the below screenshot.
      
      
       DataPower Application's mapped component
      
      
   2. We are nearly complete. At this point, we need to create the application process to run our component process against our application and mapped environment. Create it by heading to the Processes sub-tab and clicking Create Process. In my example, I have called it Deploy Basic.
      
       DataPower Application Process: Deploy Basic
      
      
   3. The Application Process, Deploy Basic, is a very simple process with a single Install Component step. Design the process as shown in the screenshot below. Once configured, Save your new Application Process.
      
       DataPower Application Process Configuration: Basic Import

6. Deploy to IBM DataPower Gateway

   1. Now that we completed the configuration of IBM UrbanCode Deploy, we can finally deploy our export.zip into IBM DataPower. To initiate the deployment, move to the main page page of your DataPower Application and click the Process Request button (small play icon) on the left-hand side of the dev environment. Here we will select the application process, Deploy Basic, and the version we wish to deploy. The environment to deploy to was chosen when you selected the Process Request button. Once the request is configured with a version and process, click Submit.
   2. IBM UrbanCode Deploy should immediately reload and show a new page containing the status of the deployment request. Here you can see the current state of the deployment, what step the process is currently on, and any failure reasonings.
      
       DataPower Application Process Request Status
      
      
   3. If the process immediately fails with a missing property exception, ensure that all properties are supplied in the locations defined above and spelling is correct in each step's configuration.
   4. Typically, we would be able open DataPower in a browser and see the changes take effect. However, because we only have a single environment and this demo imports the same configuration back into itself, there will be no changes to see. If you have a separate DataPower appliance, you can test this simple demo further by importing another export into our demo DataPower server.

Wrapping up

This concludes the Import (Basic) walkthrough for the IBM DataPower plug-in on IBM UrbanCode Deploy. In this demo, I demonstrated how to deploy a simple export.zip domain configuration into a DataPower server through IBM UrbanCode Deploy. Once configured, this process can run any number of times. In addition, if another environment is added to this application and the DataPower connection parameters are set, you can easily deploy this same export.zip to a new test or prod environment. This matches UrbanCode's general philosophy of promoting the same artifacts and deployment processes through subsequent environments. In our next walkthrough, we will take advantage of the Upload from Definition and Import (Definition) DataPower plug-in steps that a definitions file during deployment. The definitions file contains specific configuration to add, remove, or modify various properties or objects defined in the export.zip. This step is perfect for making final edits to the export.zip or small tweaks to your deployments. However, the definitions file requires specific syntax to properly achieve the defined configuration. The next blog will be fairly more advanced as I will expect you to understand a simple DataPower deployment. I will spend most of the blog explaining various examples and syntax of the definitions file.

Common Import Errors

One of the most common errors seen during any deployment is a "No Access to File" error. However, if you inspect the deployment in DataPower, everything looks to be importing as expected. This error is a SOAP specific response that is often ignored when running SOAP-based commands manually. Unfortunately, the DCM tool and plug-in interpret this response as an error. To ignore this error, use the Ignore Error Regex property present in all Import steps. Specify either a full error or regex string to ignore. You can see a working example in my configuration above and additional documentation on the GitHub Wiki page.