Creating a custom integration node setup on a virtual machine using Chef

Originally Posted by Imran Shakir | Jan 21 2014 |

In this article you will learn how to build a virtual machine that has a custom integration node setup, using the ibm_integration_bus cookbook.

The power of using infrastructure technologies such as Chef, is threefold:

• Automation of infrastructure setup, saving operational setup time (as demonstrated by the SoftLayer installation article)
• Predictability of environments created, saving operational test time
• The ability to compose different technologies in infrastructure descriptions, and execute these at the time of machine provisioning

The Hypervisor Edition of IBM Integration Bus Version 9 was released at the same time as the base product. You can use the Hypervisor Edition to deploy Integration workloads with technologies such as PureApplication Systems and IBM Workload Deployer. By using Chef, you can achieve a more flexible way of building a virtual image that includes IBM Integration Bus. The drawback of creating this flexibility is the extra time required to obtain the base operating system image, and package the final image for whichever platform you want to run it on. The benefit of using Chef over the Hypervisor Edition in this situation is the ability to customize the image with a specific integration node configuration and the option to include other software components in the image as well.

To demonstrate the flexibility of the Chef approach, consider this example, in which you have two teams, each requiring 3 integration nodes to be set up. One for development, one for test and one for production. Amongst the provisioning options available are two options (respectively displayed in the diagrams below):

• Splitting the nodes into machines by team
• Splitting the nodes by type

There are good reasons to choose either option depending on your situation. In this article, you will see that the integration nodes can still be described as Chef artifacts before that decision is made. Because of the “easily disposable” nature of virtual machines, you're therefore able to provision using one way to start off with, and then dispose of the virtual machines and provision the other way if you change your mind.

The scenario that you will walk through in this article will be a little simpler than this, involving two integration nodes and just one virtual machine. After walking through the tutorial, I will discuss how you may extend the concepts learned in the walk through to fit the scenario described above.

How do I build virtual images?

The prerequisites are:

• A virtualization provider installed on the “host” operating system. I chose VMWare Workstation 10.
• An ISO image for the “guest” operating system on which you want to install and configure IBM Integration Bus. At the time of writing, the ibm_integration_bus cookbook supports Ubuntu 12.04 and RedHat 6. I chose Ubuntu for demonstration purposes. You can download the ISO from the Ubuntu website.
• IBM Integration Bus for Developers Version 9 install image hosted on an FTP or HTTP server.

The steps are:

1. Create the virtual machine.
2. Install chef-solo on the virtual machine.
3. Get the ibm_integration_bus cookbook.
4. Create integration node descriptions as JSON documents.
5. Run the cookbook.
6. Verify the integration node setup.

The steps that I wrote are specific to VMWare Workstation 10 and Ubuntu, but I have tested this process on RedHat on VMWare Workstation 10 as well, so feel free to get in contact if you're attempting that.

Create the virtual machine

1. Open VMWare Workstation and click File -> New Virtual Machine. You will then be prompted with a wizard. Choose a Typical configuration at the first screen, and click Next.
2. At the next screen, select the second option Install disc image file (iso): and click Browse... on the right-hand side of the wizard. Navigate to the location of your operating system image, select it, and then click Open to select it for installation on the new machine. Click Next.
3. At the next screen choose a display name, a username and a password for your default machine user. I chose simply “Administrator” as my full name and “administrator” as the username. Click Next.
4. Choose a virtual machine name, allowing the machine to be identified within VMWare workstation. I chose “IIB Virtual Machine”. Optionally, you can choose a non-default location to store the virtual machine. Click Next.
5. Choose an amount of disk space (I chose 30GB). Select the default option to Split virtual disk into multiple files.
6. On the last screen, select Customize Hardware... and select Memory from the left panel. Use the slidebar to boost the memory of the machine to 4096 MB, and then click Close to finish. The virtual machine should start up and install Ubuntu. 

Install chef-solo on the virtual machine

Once the guest operating system has installed and booted, install chef-solo on it, by following all the instructions below on the guest operating system.

1. Log in using the credentials you specified from the user account creation.
2. Launch a web browser (Firefox should be on the Ubuntu sidebar by default)
3. Navigate to the web-page: http://www.getchef.com/chef/install/
4. Select your operating system, version and architecture. The options that I chose were Ubuntu 12.04 x86_64, and the site gives a command for this version.
5. Open up a terminal and run the command given on the site. You might need to install curl first, for me the two commands required were:
   sudo apt-get install curl
curl -L https://www.opscode.com/chef/install.sh | sudo bash

6. Check that the chef-solo executable has been installed by running the command below. It should give you a sensible version number:

   chef-solo –-version

Host the IBM Integration Bus for Developers image

 You have three options:

1. Host on an FTP server (the article linked to above shows how to do this on a Linux machine using vsftpd)
2. Host on a HTTP server
3. Place on the file system of the virtual machine

Any of these locations need to be accessible from the virtual machine. The quickest way to check if this is true is to open the browser in the virtual machine and type in the location of the image, which will start with ftp://, http:// or file:// depending on the hosting location you chose above. If the package starts to download, you know you've got it right!

Get the ibm_integration_bus cookbook

Now you need to set up your environment for running chef solo.

1. Create a directory in your home directory called “chef” and navigate to this directory:

mkdir chef
cd chef

2. Create a directory called “cookbooks” and navigate to this directory:

mkdir cookbooks
cd cookbooks

3. Install Git to retrieve the cookbook from the repository. Instructions for Git can be found here, but these are the steps I used to install Git:

sudo apt-get install git

4. Check the version number by using the command:

git –-version

5. Get the ibm_integration_bus cookbook. Run the following command:

Note that now running `ls` shows you the directory, ‘ibm_integration_bus‘, that contains the cookbook.

Consider now a simpler scenario than the one described in the introduction. You want to create two test integration nodes for our developer 'Tom'. Tom has been working with the IBM Integration Bus for Developers up until now, but now he is ready to put his solutions into test. He's been working on two separate projects, called 'Co-ordinated Request Reply' (CRR) and 'Batch Processing' (BP).

These two scenarios require two different integration node setups. The cookbook uses Chef a databag to describe integration node setups as JSON documents (data items). Take a look at the sample provided in the root directory of the ibm_integration_bus cookbook, named 'default_iibnode.json'. The format may look a little alien to start off with, but you'll see why by the end of the article.

For now, navigate up to your 'chef' directory, and create a directory named 'data_bags' which contains a directory named 'iib_nodes':

cd ~/chef
mkdir -p 
data_bags/iib_nodes

Each databag item in the 'iib_nodes' data bag directory corresponds to a description of an integration node that could be provisioned during your Chef run. You're going to need to create two integration nodes, so copy across the template file to your 'iib_nodes' directory, by using either a file browser or the commands:

Now you’re going to customize these descriptions to suit Tom’s needs. Use a text editor (I used gedit from the command line `gedit filename`) to edit these descriptions.

• Node Name: TOM_CRR_NODE
• Queue Manager Name: TOM_CRR_QMGR
• Two Integration Servers: “crr_server”, “additional_crr_server”
• Operation Mode: Advanced
• Default all of: Queue Manager Listener Port, Web Admin Port and HTTP Listener Port, AdminSecurity, webAdminEnabled

To achieve these properties, I edited the “crr_node.json” document to contain the content:

Integration node two requires a slightly different setup to TOM_CRR_NODE. Note that you need to set all the ports to values that are not the default, so that the two integration nodes don't clash:

• Node name: TOM_BP_NODE
• Queue Manager Name: TOM_BP_QMGR
• One Integration Server: bp_server
• Operation Mode: Advanced
• Queue Manager Listener Port: 2415
• webAdminHTTPListenerPort: 4415
• httpListenerPort: 7081

The JSON for this is:

Run the cookbook

You can now run the cookbook using chef-solo. This creates a user for IBM Integration Bus, installs the IBM Integration Bus package, tunes the operating system, and sets up the integration nodes as described in the above JSON documents.

1. Navigate back to your 'chef' directory, and create two files “solo.rb” and “solo.json”:

• The local directory path that Chef will use to cache the IBM Integration Bus image file that it acquires from wherever you've hosted it.
• The directory path for where to look for cookbooks (which will be the cookbook directory you created)
• The directory path to look for data bags (the 'data_bags' directory you created).

2. Create a JSON file, which contains the configuration for the cookbook. In the JSON file place configuration information for the cookbook, such as which recipes to run and what attributes those recipes take. In this example, you want to tell the chef-solo to run the ibm_integration_bus cookbook's default recipe. You also want to tell the ibm_integration_bus cookbook where to find the IBM Integration Bus install image, the name of the user account you wish to create, the password you wish to create, and which nodes you wish to create as part of the setup. This is described by the solo.json file:

In order to generate the “account_password”, you need to create a shadow_hash of the password. Use the command:

In this scenario:

Now execute the Chef process. Run:

Feel free to make a cup of tea or similar at this stage – the time taken to download the file will depend on the speed of your network connection. Once the IBM Integration Bus for Developers image has been downloaded by the Chef client, the rest of the process should take 20-30 minutes.

Feel free to make a cup of tea or similar at this stage – the time taken to download the file will depend on the speed of your network connection. Once the IBM Integration Bus for Developers image has been downloaded by the Chef client, the rest of the process should take 20-30 minutes.

Verify the integration node setup

1. Once the process has finished, log out of the virtual machine and log in as the user 'tom' specified.
2. Open up Integration Explorer by opening the launch menu, and typing in “IBM” and clicking on “IBM WebSphere MQ Explorer (Installation1)”.
3. Once Integration Explorer has launched, you can see that the Navigation view shows the two integration nodes that were created, each with the servers specified in the data bag items.

To verify some of the other properties, try making a REST call to one of the integration nodes. To do this I used RESTClient – which I installed on the default FireFox installation on the virtual machine. The HTTPRequest should look like:

• Method: 'GET'
• URL: 'http://localhost:4414/apiv1?depth=2'
• Header:
  • Name: 'Accept'
  • Value: 'application/json'

Note the structure of the response by clicking in “Response Body (Highlight)”. The structure here contains more information than the structure that is provided as input to the Chef script, but still has the same basic structure. A small part of the full description of an integration node can be re-created by using the Chef scripts, by using an integration node's REST response as input to the Chef script! (Although, the properties URL still needs to be called separately, and copy/pasted into the databag item.)

Conclusions

After running through these instructions you should have a working virtual machine with two integration nodes and a working installation of IBM Integration Bus. This leaves you free to utilize IBM Integration Bus with all the freedoms that virtualization gives you: easy transfer to different hardware, simple exploitation of snapshot functionality, or use in a larger IaaS provisioning framework.

This article has demonstrated the two key concepts of the Chef scripts:

• Integration node description as JSON documents: share their structure with the brokers resource model as described by the REST interface
• Ability to combine integration node descriptions to gain IBM Integration Bus environment description as Chef roles/attributes

You can see how both of these allow large amounts of flexibility in the scenario where you have two teams, each requiring a development, test and production integration node. Not only can you create the node descriptions before deciding which machines to actually provision them on, virtualization and automation means that both setups can be attempted and scrapped easily before a final conclusion is reached.

To extend the custom integration node scenario so that it applies to the original scenario, which is described with six integration nodes and two or three virtual machines, it becomes clear that one thing is necessary: a central area to store all of the information that is kept in the Chef directory.

The cookbook, the data bag items and JSON configuration can no longer be stored on just the single machine, as multiple machines (or Chef "nodes") will need access to them.

You can also store your data bag items in the Chef server. First create the data bag, and from the work station run:

knife data bag create iib_nodes

Then to create a data bag item (on Windows, use a similar text editor on Linux):

set EDITOR=notepad

knife data bag create iib_nodes bp_node

This gives you a stub of a JSON document in a text editor, into which you can copy and paste the example data bag item and edit it to your liking.

Once you have six data bag items in your Chef server, let's call them:

• team_1_dev
• team_1_test
• team_1_prod
• team_2_dev
• team_2_test
• team_2_prod

You are free to create role definitions that describe the 5 potential virtual machines that could be created in the original scenario. The only real difference between them needs to be the 'iib_nodes' attribute for the 'ibm_integration_bus' cookbook. So, in the situation in which you want to split the nodes by team, you would have two role definitions, which have different 'iib_nodes' attributes which are:

1. ["team_1_dev", "team_1_test", "team_1_prod"]
2. ["team_2_dev", "team_2_test", "team_2_prod"]

In the situation where you want to split the nodes by type, you would have three role definitions, with different 'iib_nodes' attributes, which are:

1. ["team_1_dev", "team_2_dev"]
2. ["team_1_test", "team_2_test"]
3. ["team_1_prod", "team_2_prod"]

Then to create this configuration, you would need only to bootstrap the 2 or 3 nodes, specifying those role definitions in your run list. If these are virtual machines, they're easy to destroy if you feel the decision was wrong, and you can then provision another set of virtual machines and bootstrap those instead.