Data migration between Amazon S3 and IBM Cloud Object Storage using IBM App Connect

This article describes how to use IBM App Connect to create an event-driven flow that migrates data between object storage services like Amazon S3 and IBM Cloud Object Storage.

Amazon S3 is capable of holding large amounts of data, such as backups and archives, and large individual files, such as video files, image files, and genomic data. Cloud Object Storage is a reliable, durable, and resilient cloud storage service.

Scenario

Suppose your organization needs to move all data from a partner cloud storage system like Amazon S3 to Cloud Object Storage. This task involves the transfer of a large amount of data. By using a batch retrieve operation in App Connect, you can extract all the files from an Amazon S3 bucket and upload them to a Cloud Object Storage bucket.

Before you start: This article assumes that you’ve created accounts for Amazon S3 and Cloud Object Storage.

Connecting App Connect to Amazon S3 and Cloud Object Storage

From the App Connect catalog, select Amazon S3, click Connect, and enter the requested connection details. Mandatory fields are marked with an asterisk (*). Click Connect again to connect to your Amazon S3 account. Repeat this step to connect to your Cloud Object Storage account.

Creating your flow

1. On the App Connect dashboard, click New > Event-driven flow and enter an appropriate name for your flow.
2. On the Toolbox tab, select Scheduler to add it as the trigger event of your flow.
   
   
   
   
   
   
3. Configure the scheduler by setting the type to Repeating interval, and setting it to run every hour. Also select the checkbox to make the flow run when it’s first switched on.
   
   
   
   
   
   
4. Click the plus icon (+), and from the Toolbox tab, select Batch process to add it to the flow.
5. To specify the data that you want to retrieve, select Amazon S3, expand Objects, then click Retrieve all objects to add this as the first action of the batch process.
   
   
   
   
   
   
6. Click Add condition and enter the name of the bucket from which you want to extract objects.
   
   
   
   
   
   
7. Click the plus icon (+) in the batch process to add another retrieve action. Select Amazon S3, click Show more, expand Objects, and click Retrieve object content.
8. Configure the “Retrieve object content” action by selecting the appropriate bucket, and mapping the object name to the object name of the “Retrieve all objects” action.
   
   
   
   
   
   
9. Click the plus icon (+) in the batch process to add another action. Select IBM Cloud Object Storage S3, expand Objects, then click Create object.
10. Configure the Cloud Object Storage “Create object” action by selecting the appropriate bucket, and mapping the Object name and Object content fields of the Cloud Object Storage “Create object” action to the equivalent Amazon S3 fields.
    
    
    
    Note: The date function “$now()” is appended to the object name to ensure that the object name is unique in Cloud Object Storage. If the date function wasn’t added, the flow would fail after the first successful run because the object name already exists.
    
    

The completed flow should look like this:

Test the flow

Run the flow and check that the expected data has been transferred from Amazon S3 to Object Cloud Storage.