In the latest update to IBM® App Connect we have added the capability to pause and resume a long running batch either by the user or by the system in some situations.
Batch processing is all about processing large number of records and there can be reasons for wanting to pause a batch and resume it later. This might be because you observe errors when the batch is processing and you want to try to fix those before continuing. Or it might be because you are hitting the rate limit of some applications and want to execute your batch at a different time. You might want to pause the batch to see the results of the first few records processed before deciding to let the batch run to completion. The pause and resume operations refer to the extract part of the batch.
User-controlled pause and resume
Let’s take a flow with two batches that we have built earlier and see how we can control the batches.
Check the flow for the identifier of each batch (for example Batch Process or Batch Process 2).
From the dashboard, find the tile for your flow. In its three-dots context menu, you will find the options to view batches.
In the Show batches dialog box, you can see the two batches and can choose a new action to Pause retrieving.
After choosing the Pause retrieving operation, the table is updated to show that the retrieve operation is paused:
And on the dashboard, you can see the information about the batch.
From the Batch status dialog box and the row that’s associated with that particular batch, you can now resume the batch and it will go back to the running state, continuing to extract records.
If you check the logs from the hamburger menu, you will see entries like these for the operations that you’ve performed
System pause and resume
There are some situations in which the extract process fails because of errors. In this case, App Connect automatically pauses the batch and tries to resume it after a short interval in the hope that the problem that caused the error has been fixed. When this happens, the flow tile on the dashboard will look like this:
And the log entries will be like this:
If the auto-resume does not succeed, App Connect will auto-pause the batch again and then try to resume, until the batch times out or you decide to stop it.
A few things to note:
- Stopping the flow will stop all batches, including paused batches.
- The pause and resume operations apply to the extracting of the messages from the source system, not to the processing of those messages. There is no option available to pause or resume the processing of the items in the batch.
- Certain batch errors are not recoverable. Say a password to access one of the applications that’s used by the batch process has been changed. To update the account, you’ll need to stop the flow and update the account, and this will stop the batch(es) for that flow.
- If you have multiple batches running in your flow, the tile will show the most serious problem associated with one of your batches (if any). So it’ll show an auto-paused batch due to an error if there is one. If no batch is auto-paused, the tile will show if you have paused a batch. If all is good, you will see the “Batch in progress” message.
With this feature, you can have more control over your batches.