Working with IBM® MQ Managed File Transfer (MQMFT)

Fri October 15, 2021 12:24 PM

Stephanie Wilkerson

File / Message transfer in Enterprise Systems using IBM® MQMFT on top of an existing MQ Infrastructure (Cluster & Distributed)

This article demonstrates how to transfer Files / Messages using MQ Managed File Transfer across Enterprise Systems using an existing IBM® MQ network Infrastructure (both Cluster & Distributed).

Overview

Skill Level: Advanced

Working knowledge on IBM® MQ

Ingredients

IBM® MQ Managed File Transfer (MQMFT) transfers files between systems in a managed and auditable way, regardless of file size or the operating systems used. We are going to define some of the import concepts related MQMFT before proceeding with the actual implementation.

Concepts and Terminologies:

The Coordination, Agent, and Command Qmgrs are roles which runs on top of a given Qmgr(s). A single Qmgr can fulfill the role of all three, or you can have a queue manager dedicated to each role. The choice depends on your existing MQ infrastructure and the expected load on the network.

Agent
MQMFT agent is a JVM process that runs on a machine and performs file transfers to and from other agents. In order to transfer files to or from a machine using MQMFT, you must have an agent running on that machine. Every agent connects to a IBM MQ Qmgrs and uses MQ to communicate with other agents. The queue manager that an agent connects to fulfills the agent queue manager role, and will have a number of queues defined on it for the agent. An agent will connect only to its agent queue manager, which it does using bindings or client connection modes.

Agent Queue Manager
A Qmgr that hosts an Agent's queues. Each Agent needs a number of Queues defined to be able to function.

Coordination Queue Manager
A Qmgr that broadcasts audit of file transfer and acts as a central location.It also gathers information about all the agents in the network at a single point where it can be queried. All agent Qmgrs in the MQ network must have connectivity to the coordination queue manager to route appropriate information there.

Command Queue Manager
Submitting commands to an Agent (such as requesting a transfer) involves sending an MQ message to the agent's command queue on the Agent Qmgr. The Command Qmgr is the queue manager used by the command-line commands to connect to, in order to submit these command messages to Agents. IBM MQ routes the messages from the command queue manager to the appropriate Agent queue manager. The Command queue manager must have connectivity to all Agent Qmgrs.

How does IBM MQ Managed File Transfer work ? Read here.

The following Components need to be installed to be able use use MQMFT on Windows.

MFT Packages for Linux Platform.

Step-by-step

IBM® MQMFT over Cluster Network

IBM® MQ v8.0 & Above

Platform: Windows

Introduction: In this article we are going to demonstrate the implementation of MQMFT over an existing MQ Cluster Infrastructure. The below diagram represents our MQ Managed File Transfer Implementation for a given Organization having a Cluster ‘CLUS’ network. QM1 & QM2 are the Full Repository Qmgrs & the remaining Qmgrs are the Partial Repository. COLO (Coordination & Logger) is acting as the Coordination Qmgr for this MFT network. It also hosts the Logger function. The other four Qmgrs i.e. QM1, QM2, QM3 & QM4 have been configured as the Command Qmgrs & also hosts the Agent Processes.

We are going to use the below diagram as the Reference Architecture for our MQMFT Implementation. Using this scenario, we are going to send files across Agent Qmgrs in the cluster network.
Create a MQ Cluster

In this Section, I are not going to create a MQ Cluster from scratch. Steps on how to create a MQ Cluster has already been mentioned in a separate but related article on IBM® MQ, an Enterprise Messaging Backbone in a True Sense! Hence, I am not going to repeat the Steps for Cluster creation & configurations here. This is how my system looks like after installation of MQMFT. The command to diplay the version of MQMFT is fteDisplayVersion as shown in the below screenshot.

We need to define a dedicated SVRCONN Channel through which the Coordination, Agent & Command processes are going to Interact with the respective Qmgrs in either Client or Binding mode. In our case, the Interaction will be in Binding mode because this entire exercise has been demonstrated in my local machine.

runmqsc QM1
#############
DEFINE CHANNEL(MFT.SVRCONN)

Similarly, repeat the same step for other four Qmgrs in the Cluster i.e. COLO, QM2, QM3 & QM4.

Note: We have disabled the default MQ Security features in order to ensure no Security related errors while configuring the MFT. We will enable these security features once performed successfuly MFT configurations.

ALTER QMGR CHLAUTH(DISABLED) ===> Disable MQ Channel Security (MQ v7.5 & above)

ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL)
REFRESH SECURITY TYPE(CONNAUTH) ===> Disable ID/Password Security (MQ v8.0 & above)

Final Setup Configuration: MQ Channels from QM1 (FR) Qmgr.

Final Setup Configuration: MQ Channels from QM2 (FR) Qmgr.
Setup the Coordination Qmgr (COLO)

In this Section, we are going to setup the Coordination Queue Manager on top of COLO (PR) Qmgr as shown in the above Reference Architecture Diagram in Section 1.

Generic Command: fteSetupCoordination -coordinationQMgr <QmgrName> -coordinationQMgrHost <Hostname> -coordinationQMgrPort <Port No.> -coordinationQMgrChannel <SVRCONN Channel>

We have run the COLO.mqsc file against the Qmgr to create the necessary Queue objects required by Coordination Process.

Generic Command: runmqsc QmgrName <Qmgr.mqsc > Qmgr.mqsc.out

Location: C:/Program Files (x86)/IBM/WebSphere MQ/mqft/config/COLO
Setup the Logger (COLO)

Again, we are going to use the same Qmgr COLO (Coordination-Logger) to configure the Logger Process. Logger can be used as a DataBase or a Flat file to store information on historical account of file transfers. In our case, we have configured Logger to use a Flat file & limit its size to 5 MB.

Generic Command: fteCreateLogger -loggerType FILE -loggerQMgr <QmgrName> -fileLoggerMode LINEAR -fileSize 5MB LOGGER_<QmgrName>

LOGGER_COLO_create.mqsc file has been created on successful completion of fteCreateLogger command. We have run this .mqsc file against the Qmgr to define the necessary MQ Objects required by Logger for its operations.

Generic Command: runmqsc QmgrName <LOGGER_(Qmgr)_create.mqsc > LOGGER_(Qmgr)_create.mqsc.out

Location: C:/Program Files (x86)/IBM/WebSphere MQ/mqft/config/COLO/loggers/LOGGER_COLO

Logger Commands
##################
fteStartLogger LOGGER_COLO
fteStopLogger LOGGER_COLO
ftsShowLoggerDetails
Configure Agents & Command Qmgrs

In this Section, we are going to create the Command properties file along with Agent Process for a given Qmgr QM1.

Command 1: fteSetupCommands -connectionQMgr QM1 -connectionQMgrHost 127.0.0.1 -connectionQMgrPort 1414 -connectionQMgrChannel MFT.SVRCONN

Command 2: fteCreateAgent -agentName AGENT_QM1 -agentQMgr QM1 -agentQMgrHost 127.0.0.1 -agentQMgrPort 1414 -agentQMgrChannel MFT.SVRCONN

command.properties
#####################
#
#Sat May 27 11:02:37 EDT 2017
connectionQMgrChannel=MFT.SVRCONN
connectionQMgrPort=1414
connectionQMgrHost=127.0.0.1
connectionQMgr=QM1

######################################

We have run the AGENT_QM1_create.mqsc file against the given Qmgr QM1 to create necessary MQ Objects required by Agent Process AGENT_QM1.

Generic Command: runmqsc QmgrName <AGENT_(Qmgr)_create.mqsc > AGENT_<Qmgr>_create.mqsc.out

agent.properties
################
#
#Sat May 27 11:02:37 EDT 2017
connectionQMgrChannel=MFT.SVRCONN
connectionQMgrPort=1414
connectionQMgrHost=127.0.0.1
connectionQMgr=QM1
enableQueueInputOutput=true

##############################

Location: C:Program Files (x86)IBMWebSphere MQmqftconfigCOLOagentsAGENT_QM1

Now, since the Command & Agent Process has been setup, we need to start the Agent using the command: fteStartAgent AGENT_QM1

Similarly, repeat the same Steps of Command & Agent Process configuration on other Qmgrs i.e. QM2, QM3 & QM4.

Agent Commands
#################
fteStartAgent AGENT_QM1
fteStopAgent AGENT_QM1
fteShowAgentDetails AGENT_QM1
fteListAgents
ftePingAgent -m QM1 -w 10 AGENT_QM1

Once, all the Agent Processes has been configured for remaining Qmgrs in the Cluster except (COLO which is a dedicated Coordination & Logger Qmgr), list down the Agent processes using the below command: fteListAgents

command.properties ==> For All Qmgrs in the Cluster.

Now, it time to ping the FTE Agents in the Cluster network. Ensure that all the Cluster Channels are running before issuing FTE Ping command. Observer the response time in seconds marked in red.

Another Output of ping FTE Agents from MQ Explorer. Ready Status signify that the Agent is ready to accept request for File transfer.

Log File Location for Agents & Logger Process: C:/Program Files (x86)/IBM/WebSphere MQ/mqft/logs/COLO
File to FileTransfer

In this Section, we are going to demonstrate the File Transfer (using MQ Explorer wizard) between various Qmgrs (Agents) in the Cluster network using the sequence of screenshots as shown below. We can also initiate the same transfer using Command Line Interface.

1. Initiate a new File Transfer by Right Click on Managed File Transfer option as shown below & select New Transfer…

2. Select the Source & Destination Agents i.e. the Souce System from where you are going to pick up the file & the Destination System where you are going to send the file.

3. Click on Add button to select the Source system file & other associated transfer parameters required to complete File Transfer.

4. Here, select the various options as per your requirement for File Transfer. Some of the important parameters are marked in the below screenshot. In this case, we are going to transfer the file named POC.bat from the C drive of the system where AGENT_QM1 is running & place the same on the Desktop location of the other Agent AGENT_QM4.

5. Final Transfer Configuration screenshot.

6. You have the option of scheduling a Job to transfer files on specific date/time along with other variable configuration parameters as shown below.

7. The below screenshot show the file POC.bat has been successfully transferred from Souce to Destination Agents marked in Red.

Alternate Command: fteCreateTransfer -sa AGENT_QM1 -sm QM1 -da AGENT_QM4 -dm QM4 -df “C:UsersIBM_ADMINDesktopPOC.bat” “C:POC.bat”
Message to File Transfer

In this Section, we are going to put some messages in a Local Queue named POC defined under QM1. Using MQMFT, we are going to trasfer these 10 messages into a single File called TEST.txt to the Destination machine.

1. Here, I have put 10 messages into the Local Queue POC.

2. In this Step, I have Selected the End point Agents i.e. Source & Destination be to AGENT_QM1 & AGENT_QM2. Selected the source to be Queue POC & the destination to a Text file by the name TEST.txt. As part of this operation, all the messages will be moved into the text file & place the same on Desktop.

3. Final configuration screenshot before actual transfer.

4. All 10 Messages has been successfully transferred as a Text file as shown below in the screenshot.

5. Final output of the TEXT.txt file on the Desktop of the Destination Agent.
File to Message Transfer

In this Section, we are going to send a Text file TEST.txt as a Message to a Remote Agent Qmgr.

1. Here, the Source Agent is AGENT_QM4 & the Destination is AGENT_QM1. From the Souce machine, I am send a file named TEST.txt to a Local Queue POC on the Remote Agent machine as shown in the below screenshot.

2. The below screenshot shows the Text File has been successfully sent across machines & put into the Local Queue POC as message to remote agent (AGENT_QM1) Qmgr QM1.

3. The below screenshot show the souce file TEST.txt & its correspoding Message into the Local Queue POC.
IBM® MQMFT over Distributed Network

In this distributed setup, we are going to use the same Qmgr COLO as the Coordination & Logger Qmgr as created in Section 3 & 4 of this article. Hence, in this section we won’t repeat the steps but create the remaining MQ Connectivity configurations between the Qmgrs in this distributed network as per the below diagram. QMA, QMB, QMG and COLO are the four Qmgrs, used to demonstrate the File Transfer mechanism using MQMFT. COLO is dedicated as the Coordination & Logger Qmgr. Whereas the remaining three Qmgrs has been setup as the Agent & Command Qmgrs.

As per the MQMFT requirement, all Agents must be registered with the COLO i.e. Coordination Qmgr in a MQMFT network. Hence, our first task would be to define normal Sender / Receiver Channels (Bi-directional) between COLO & other three Qmgrs i.e. QMA, QMB & QMG. Also, if we want to transfer files between Agent Qmgrs, you need to have MQ Channel connectivity in the first place. Hence, in the next section we are going to complete all the basic MQ configurations required for Managed File Transfer.
Create a Distributed Network

In this Section, I are not going to create a Distributed MQ Network from scratch. Steps on how to create a similar setup has already been mentioned in a separate but related article on IBM® MQ, an Enterprise Messaging Backbone in a True Sense! Hence, I am not going to repeat the same steps here. Here is what we have to do from MQ perspective:

Step 1 (Required to register Agents with Coordination Qmgr)
################################################
COLO <=== SDR / RCVR Channel ===> QMA
COLO <=== SDR / RCVR Channel ===> QMB
COLO <=== SDR / RCVR Channel ===> QMG

Step 2 (Required to transfer files between Agent Qmgrs)
#############################################
QMA <=== SDR / RCVR Channel ===> QMB
QMA <=== SDR / RCVR Channel ===> QMG
QMB <=== SDR / RCVR Channel ===> QMG

Besides, we need to define a dedicated SVRCONN Channel through which Agents & Commands processes are going to Interact with the respective Qmgrs in either Client or Binding mode. In our case, the Interaction will be in Binding mode because this entire exercise has been demonstrated in my local machine.

runmqsc QMA
##############
DEFINE CHANNEL(MFT.SVRCONN)

Similarly, repeat the same step for other two Qmgrs in the network i.e. QMB & QMG. (Already completed this activity for COLO Qmgr in Section 2)

Note: We have disabled the default MQ Security features in order to ensure no Security related errors while configuring the MFT. We will enable these security features once performed successfuly MFT configurations.

ALTER QMGR CHLAUTH(DISABLED) ===> Disable MQ Channel Security (MQ v7.5 & above)

ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL)
REFRESH SECURITY TYPE(CONNAUTH) ===> Disable ID/Password Security (MQ v8.0 & above)

Final Setup Configuration: MQ Channels from COLO Qmgr

Final Setup Configuration: MQ Channels from QMA Qmgr
Configure Agents & Command Qmgrs

Now, we are going to create the Command properties file along with Agent Process for Qmgr QMA.

Command 1: fteSetupCommands -connectionQMgr QMA -connectionQMgrHost 127.0.0.1 -connectionQMgrPort 1430 -connectionQMgrChannel MFT.SVRCONN

Command 2: fteCreateAgent -agentName AGENT_QMA -agentQMgr QMA -agentQMgrHost 127.0.0.1 -agentQMgrPort 1430 -agentQMgrChannel MFT.SVRCONN

Once the Agent has been created, we have run the AGENT_QMA_create.mqsc file against the given Qmgr QMA to create necessary MQ Objects required by Agent Process AGENT_QMA as shown below in the screenshot.

Generic Command: runmqsc QmgrName <AGENT_(Qmgr)_create.mqsc > AGENT_<Qmgr>_create.mqsc.out

Location: C:/Program Files (x86)/IBM/WebSphere MQ/mqft/config/COLO/agents/AGENT_QMA

Similarly, repeat the same steps for Command & Agent process configuration on QMB & QMG Qmgrs.

Agent Commands
#################
fteStartAgent AGENT_QMA
fteStopAgent AGENT_QMA
fteShowAgentDetails AGENT_QMA
fteListAgents
ftePingAgent -m QM1 -w 10 AGENT_QMA

Once, all the Agent Processes has been configured for remaining Qmgrs except (COLO which is a dedicated Coordination & Logger Qmgr), list down the Agent processes using the below command: fteListAgents

Verify the Agent Connectivity using Ping Command from MQ Explorer:

command.properties ==> For All Qmgrs in the Distributed Network.
File / Message Transfer

Now, its time to send files across Agent Qmgrs within the Distributed Network. The procedure is exactly same as we have done in the Cluster setup. Select the Source & Destination Agents, Select File, Transfer Mode etc. as shown in the below screenshot. In this Case, I am going to send the file POC.bat from C drive of the Source Agent to Desktop of the Destination Agent as POC1.txt.

1. File to File Transfer between AGENT_QMB & AGENT_QMA

Click on the Finish button. If all the configurations are fine, then File transfer will be successful as shown in the below screenshot.

2. Message to File Transfer: In this Case, I am putting a message into a Local Queue called POC123 defined under Qmgr QMA as shown in the below screenshot. Using the MFT Agents, I am going to send this message as a Text file to the destination system.

Select the Source & Destination Agents. Select the Queue POC123 in the Souce Agent Qmgr, transfer type as Text transfer & specify the Destination path & file name POC123.txt as shown in the below screenshot.

The below screenshot shows that the message has been successfully transferred to the Destination System as a Text file POC123.txt.
Conclusion

Thus, we have demonstrated the technical implementation of IBM® MQ Managed File Transfer (MQMFT) on top of an existing MQ Infrastructure either in the form of a Cluster or Distributed setup. MQMFT is one of the Advanced MQ features and hence it is essential to have some basic understanding of Agents, Coordination, Command & Logger components before proceeding with actual configuration. Since, MFT is basically using IBM® MQ as it underlying Infrastructure, we can apply all MQ Security features on top of MQMFT as well. For example, we can configure the MQ Channels to use SSL/TLS Certificates between Agent Qmgrs. In this article, I have used a series of related Screenshots as a step by step guide on how to configure MFT Agents, Coordination Qmgr, Logger etc. to simplify the process of MQMFT implementation.
Related Topics & References

1. http://www-01.ibm.com/support/docview.wss?uid=swg27041613&aid=1

2. http://www.mqug.org.uk/downloads/201307/201307%20-%20WMQ03%20-%20Introduction%20to%20WebSphere%20MQ%20Managed%20File%20Transfer.pdf

3. http://mqtechconference.com/sessions_v2014/MQTC_Intro_t_%20MQ_MFT.pdf

4. https://www.ibm.com/products/mq

5. Managed File Transfer Solutions

6. https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.wmqfte.doc/wmqfte_intro.htm

7. https://www.ibm.com/developerworks/websphere/library/techarticles/1003_phillips/1003_phillips.html