IBM MQ Managed File Transfer (MFT) transfers files between systems in a managed and auditable way, regardless of file size or the operating systems used.
Amazon Web Services (AWS) is an on-demand cloud computing platform which can be utilized by individuals/enterprise on a pay-as-you-go basis. Within AWS, there is something called Amazon Simple Storage Service (Amazon S3). This is a storage for the internet. You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web. Every object/file in S3 is stored in a bucket.
Do you know IBM MQ MFT Agent can also be used to communicate with AWS S3?
To answer this, there are two basic facts we must know on either sides,
1) Since Amazon S3 is a file storage and MFT is all about transfers of files to & fro to such storages, S3 also can also be used as one of source or a destination for MFT. This is particularly possible via MFT's protocol bridge agents.
2) The Protocol bridge agents are a special type of MFT agents which enable MQMFT network to access files stored on a file server outside your MQMFT network, either in your local domain or a remote location. File servers can use the FTP, FTPS, or SFTP network protocols, all of which are supported by the PB agents.
After having above points in mind, anyone trying to achieve the interconnect between the two, one must
understand AWS Transfer Family.
The AWS transfer family, in brief, enables users to transfer files either way to AWS storage services.
Components required for achieving this end-to-end flow:
Among storage services, as we know the point of interest for us is S3 bucket. Below are components which are necessary,
1) S3 Bucket
2) SFTP/FTP/FTPS server in AWS transfer family
3) Server needs to be configured with a logging role of an user having IAMs - AmazonS3FullAccess, AWSTransferConsoleFullAccess
4) An external user for this particular server and corresponding key for access need to be created. This is required to be used to access the server from outside of AWS network to connect to the S3 bucket of our choice
5) On-premise Protocol bridge agent***
The flow looks something like below,
AWS S3 bucket <--> AWS transfer family <--> IBM MQ MFT Protocol Bridge Agent
*** - Note: Once the S3 bucket & server are setup in AWS transfer family, the endpoint-URL available for the server needs to be copied and configured in the PB agent's properties file as detailed in this link. The subsequent user/key combination has to be provided via ProtocolBridgeCredentials.xml .Post this the entire system would be ready for transfers.
The file transfer flow can be represented by below diagram,
Few key things to be kept in mind while configuring entire system:
1) During the creation of S3 bucket, couple of points have to be taken care of, which are,
a) The bucket's ACLs have to be enabled as below. This is needed for the mft to write files to S3.
b) The "Block all public access" check has to be unchecked. Its shown below,
3) On the MFT's side, once entries are made to ProtocolBridgeProperties.xml and ProtocolBridgeCredentials.xml using the endpoint-URL & User, created in previous steps, the MFT agents will be ready for transfers. Below is a sample fteCreateTransfer command to run the transfer,
fteCreateTransfer -sa SourceAgent -sm QM1 -da PBAgent -dm QM1 -de overwrite -dd "awsServer://mqmftbucket//dest//" "C:\Users\Administrator\Desktop\src\*.*"
Make sure you have escaped the AWS S3's destination directory path, i.e., like this --> //, as seen in the above command fteCreateTransfer.