App Connect

So you've been authoring some batch flows in the IBM App Connect Enterprise as a Service Designer. You've used the batch status dialog to observe your batches running and completing, and maybe even stopped and started them mid-process, all from within the UI. But how do you now deploy this for production, integrate it into your CI/CD pipeline, and monitor how it's behaving? Read on to learn more.

Hopefully you're familiar with some of our previous blogs. There's ACEaaS - Technical Overview where Martin gives an overview of IBM App Connect Enterprise as a Service and explains how you use the Dashboard UI to deploy flows during your Trial or when subscribed to the VPC Hours plan. Ananya and Bijoy explain how this process is streamlined for the Flow Run plan in Introducing Resource Unit (Flow Run) plan. You'll also know that there is a Public API that you can use to achieve tasks programmatically rather than with the use of User Interfaces, allowing you to automate them in a CI/CD pipeline.

Finally, you'll have an understanding of batch process nodes and possibly tried them out for yourself using the Designer Authoring tools thanks to Sam's Introducing Batch Flows, so let's pick things up from there.

Before you start

Some of the steps in this blog assume you have a batch flow in Designer that you are ready to deploy to an integration runtime. If you do not have such a batch flow then please create one by following the steps in Introducing Batch Flows before beginning.

Getting started

First up let's ensure we've got a set of credentials to use the Public API, as we'll be using that for all sorts of tasks. If you followed Adam's blog post you may already have a set of these you can use.

1. From the App Connect Dashboard for your instance, navigate to the Settings panel and click the "Public API credentials" tab. 

2. Click on the "Generate" button and enter a name when prompted. This will give you a client ID and a client secret. Keep these safe and consider them both sensitive.

3. From the same page, you will see a link to generate an API key. You will need to follow the steps on that page as well to get an API key.

4. Once you have the client id, client secret, and API key, these are not expected to change frequently. The current expiry date at the time of this post is two years. 

5. You will use these three pieces of information as well as your App Connect instance ID to generate an access token that you can use to interact with App Connect. 

We'll be using the curl command as it's readily available, but feel free to use any API tool you wish.

As we're using curl we'll set some variables in the environment. You may not wish to set your credentials in this way, so if this is the case just substitute the values when you see them.

In the following, if your url was, for example https://abcd1efgh-designer.p-lon-c1.appconnect.automation.ibm.com, then <region> would be 'p-lon-c1' and <instance short id> would be 'abcd1efgh'.

export appConEndpoint=https://api.<region>.appconnect.automation.ibm.com
export appConInstanceID=<instance short id>
export appConClientID=<client id>
export appConClientSecret=<client secret>
export appConApiKey=<api key>

Now make a request for and store an access token. This will last for around 12 hours. You may need to repeat this step if it expires.

curl --request POST --url "$appConEndpoint/api/v1/tokens" --header "X-IBM-Client-Id: $appConClientID" --header "X-IBM-Client-Secret: $appConClientSecret" --header "content-type: application/json" --header "x-ibm-instance-id: $appConInstanceID" --data '{"apiKey":"$appConAPIKey"}'

export appConToken=<access_token>

Deploying your flow

Production flows must be deployed as integration runtimes. I'll cover how to do this for a trial or the VPC hours plan. If you're on the Flow Runs plan you can do this from the 'Manage' tab of the Designer UI rather than exporting to the Dashboard UI, and you won't need to create an Accounts configuration.

If you followed our introduction to batch blog post you'll have a flow in your Designer UI called 'QualiTea Batch Flow'. Export the flow as a BAR file to save it to your file system.

Next switch to the Dashboard UI and click the 'Deploy integrations' shortcut tile.

A small sized integration should be more than sufficient for our needs so choose that, and on the next page use the top box to drag and drop or click to select the .bar file which you exported from the Designer UI. Once successful it will automatically be selected as the 'File to be imported'.

We'll need a configuration of type Accounts which contains the connection details for the applications of your flow. Use the 'Add account' button to add additional accounts. Once created make sure it is selected for inclusion in your integration runtime.

Click Next again. On the final 'Properties' step let's give the integration runtime a more appropriate name such as 'qualitea-batch-ir'. Other settings on this page are beyond the scope of this blog post, but you can read about them in the documentation.

Click 'Create' and you'll be taken to the Runtimes page where you'll see a tile representing your integration runtime. It will probably be in a Pending state, so let's refresh the page until it becomes Ready.

At this point you may want to check the Logs view, and the output in your third party applications to see if the batch flow is operating as expected. If it is we'll continue to get a more detailed view of the batch status.

Getting the status of batches

Let's start by getting all of the batches for the integration runtime. Since we only have one flow running here this would be similar to opening the batch dialog in Designer, but if there were other flows running we'd see those here too. You'll see batches which have completed within the last six hours.

curl --url "$appConEndpoint/api/v1/integration-runtimes/qualitea-batch-ir/batches" --header "Authorization: Bearer $appConToken" --header "X-IBM-Client-Id: $appConClientID" --header "Content-Type: application/json" --header "x-ibm-instance-id: $appConInstanceID"

{"batches":[{"id":"d10514bc-f70d-46a0-b822-9d184c5841cb","flowId":"QualiTea_Batch_Flow","state":"complete","start":1705063425818,"end":1705063445471,"expiry":1707655425818,"extract":{"recordsExtracted":21},"recordsProcessed":{"success":21,"error":0,"canceled":0,"total":21}}]}

You can process the response however you like, read the raw data, or parse and display it with your own Ops tools. I like to pipe the output into jq, a lightweight and flexible command-line JSON processor, to format it into something which resembles the Designer batch status dialog that we're now familiar with.

curl --url "$appConEndpoint/api/v1/integration-runtimes/qualitea-batch-ir/batches" --header "Authorization: Bearer $appConToken" --header "X-IBM-Client-Id: $appConClientID" --header "Content-Type: application/json" --header "x-ibm-instance-id: $appConInstanceID" | jq -r '["id","status","start","end","expiry","retrieved","processed","succeeded","failed","canceled"], ((.batches | sort_by(.start) | reverse)[] | [.id, .state, ((.start // 0) / 1000 | todate), ((.end // 0) / 1000 | todate), ((.expiry // 0) / 1000 | todate), .extract.recordsExtracted, .recordsProcessed.total, .recordsProcessed.success, .recordsProcessed.error, .recordsProcessed.canceled]) | @tsv' | column -ts$'\t'

id                                    status    start                 end                   expiry                retrieved  processed  succeeded  failed  canceled
d10514bc-f70d-46a0-b822-9d184c5841cb  complete  2024-01-12T12:43:45Z  2024-01-12T12:44:05Z  2024-02-11T12:43:45Z  21         21         21         0       0

Operating on a single batch

When it comes to these tasks, you'll need to specify both the batch ID and a flow ID. The flow ID will be based on the flow name, or you can see it's value when querying all batches.

Getting the status of a single batch:

curl --url "$appConEndpoint/api/v1/integration-runtimes/qualitea-batch-ir/batches/d10514bc-f70d-46a0-b822-9d184c5841cb?flow_id=QualiTea_Batch_Flow" --header "Authorization: Bearer $appConToken" --header "X-IBM-Client-Id: $appConClientID" --header "Content-Type: application/json" --header "x-ibm-instance-id: $appConInstanceID"

{"id":"d10514bc-f70d-46a0-b822-9d184c5841cb","state":"complete","nodeName":"Batch process","start":1705063425818,"flowId":"QualiTea_Batch_Flow","recordsProcessed":{"success":21,"error":0,"canceled":0,"total":21},"extract":{"state":"complete","recordsExtracted":21,"pausedTime":0,"pausedCount":0},"end":1705063445471}

This wouldn't work for our completed batch, but say we had an example where it was extracting a larger number of records. We could, if necessary, pause that process temporarily, and resume it subsequently:

curl --request POST --url "$appConEndpoint/api/v1/integration-runtimes/qualitea-batch-ir/batches?flowId=QualiTea_Batch_Flow" --header "Authorization: Bearer $appConToken" --header "X-IBM-Client-Id: $appConClientID" --header "Content-Type: application/json" --header "x-ibm-instance-id: $appConInstanceID"

Again, this wouldn't apply to our completed batch, but we could, if we so choose, stop it completely. Once it is stopped, however, it cannot be resumed.

curl --request POST --url "$appConEndpoint/api/v1/integration-runtimes/qualitea-batch-ir/batches/50619169-b1aa-42b7-9245-612fdb6d09cf/stop" --header "Authorization: Bearer $appConToken" --header "X-IBM-Client-Id: $appConClientID" --header "Content-Type: application/json" --header "x-ibm-instance-id: $appConInstanceID"

Cleaning up

We should delete our integration runtime so that it does not continue to contribute towards our VPC hours allowance unnecessarily.

You may also wish to remove the Accounts configuration from the Configuration page, and the .bar file from the BAR files page.

Further reading

API overview.

More documentation for batch is available through the IBM Documentation.