DataPower

 View Only
  • 1.  Failed to establish a backside connection error

    Posted Mon November 27, 2023 02:19 PM

    Good afternoon colleagues.

    In recent days, problems have begun to arise in the interaction of our productive DataPower with the queue manager deployed on the IBM MQ Appliance. Errors of the following type began to appear on the DataPower side:

    [0x808000f2][XXX4][error] mpgw(XXX4): trans(1018701471)[error][ip_client_1] gtid(e6b99f89655f4d233cb8269f): XSLT custom log message '  sender: TESTXXX400        signer: TESTXXX400       action:Receive   msg:Код помилки: 1010 Пояснення: Failed to establish a backside connection'
     [0x01130006][mpgw][error] mpgw(XXX4): trans(1018701471)[error][ip_client_1] gtid(e6b99f89655f4d233cb8269f): Failed to establish a backside connection

    [0x80e0012b][mpgw][error] mpgw(XXX4): trans(1018701471)[ ip_client_1] gtid(e6b99f89655f4d233cb8269f): Backside header ('N/A') failed to parse due to: Failed to establish a backside connection, URL: dpmq://<Queue Manager>?ReplyQueue=TEST;GMO=2;TimeOut=2;ParseProperties=on

    On the IBM MQ Appliance side, errors are generated stating that the initiator of the connection termination was IBM DataPower.

    We could not find in the IBM DataPower documentation an error message with code 1010 that occurs when connecting to an SSL queue manager on an IBM MQ Appliance. It is also not clear why DataPower broke the existing connection with the queue manager on the IBM MQ Appliance.



    ------------------------------
    Andrii Kushneryk
    ------------------------------


  • 2.  RE: Failed to establish a backside connection error

    Posted Mon November 27, 2023 04:41 PM

    Hi Andrii,
    Given this is a production environment, have you opened a PMR/support ticket?

    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 3.  RE: Failed to establish a backside connection error

    Posted Tue November 28, 2023 03:49 PM

    Hello Steve.

    Connection error with MQ Queue Manager occurred in a production system. We have a queue manager running on IBM MQ Appliance, so this is the first time I have encountered an error related to the disconnection of an existing connection to the queue manager. While maintaining other information systems that work through the MQ client, the loss of connection to the MQ queue manager caused by an error in the application software causes error messages on the MQ side of another type.

    Since it is now difficult to find documentation that describes the specifics of DataPower's interaction with the MQ manager, as well as any additional information on this topic, I would like to receive links to documentation or presentations on this topic.

    So the error is not of a periodic nature, we have not opened a case in IBM Support so have not yet been able to understand the nature of the occurrence of such an error.



    ------------------------------
    Andrii Kushneryk
    ------------------------------



  • 4.  RE: Failed to establish a backside connection error

    Posted Tue November 28, 2023 05:02 PM

    Hi Andrii,
    I queried our support MQ SME who provided the following response:

    The source of connection broken from DP MQ client (reason code 1010) is not clear. It can happen if cache-timeout is not configured whose value should be less than the qmgr keepAlive. Since qmgr is running in the MQ appliance, it will be useful to collect packet traces from DP and MQ traces from qmgr side to debug this issue. You can also check the qmgr side error logs and see what error code is visible with respect to DP error code of 1010. If there is any IP firewall between DP and MQ appliance, you should also check the connection idle timeout of the IP Firewall to synchronize the timeous as follows: DP mq-qm cache-timeout < IP Firewall idle timeout < MQ qmgr keepAlive timeout.

    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 5.  RE: Failed to establish a backside connection error

    Posted Fri December 01, 2023 03:22 PM

    Hello Steve,

    To answer the question regarding the configuration and pool size for creating connections to MQ, I took screenshots of our queue manager connection configuration.

    This is the second page of the configuration that deals with connections


    ------------------------------
    Andrii Kushneryk
    ------------------------------



  • 6.  RE: Failed to establish a backside connection error

    Posted Mon December 04, 2023 02:15 PM

    Thanks Steve!



    ------------------------------
    Thomas Burke
    ------------------------------



  • 7.  RE: Failed to establish a backside connection error

    Posted Wed November 29, 2023 05:05 PM

    Hello Andrii,

    The source of connection broken from DP MQ client (reason code 1010)  is not clear. It can happen if cache-timeout is not configured whose value should be less than the qmgr keepAlive. Since qmgr is running  in the MQ appliance, it will be useful to collect packet traces from DP and MQ traces from qmgr side to debug this issue. You can also check the qmgr side error logs and see what error code is visible with respect to DP error code of 1010.

    If there is any IP firewall between DP and MQ appliance, you should also check the connection idle timeout of the IP Firewall to synchronize the timeous as follows: DP mq-qm cache-timeout < IP Firewall idle timeout < MQ qmgr keepAlive timeout.



    ------------------------------
    Chin Sahoo
    ------------------------------



  • 8.  RE: Failed to establish a backside connection error

    Posted Fri December 01, 2023 03:04 PM

    Hello Chin.

    I have analyzed the error log of the MQ queue manager. The result of the analysis is that the DP itself broke the connection to the MQ by force.

    The first mistake on the MQ side was:

    <11>Nov 23 15:01:34 [0x8d009665][qmgr][error] qmgr(Queue Manager): [ip_address_DP]: AMQ9665E: SSL connection closed by remote end of channel 'CH.SVR.SSL'. [ArithInsert1(420), CommentInsert1(CH.SVR.SSL), CommentInsert2(gsk_secure_soc_read), CommentInsert3(ip_address_DP)]

    After that, the following error appeared:

    After that, there was already an attempt on the part of MQ to write to a channel that was no longer there:

    <11>Nov 23 15:01:34  [0x8d009206][qmgr][error] qmgr(Queue Manager): [ip_address_DP]: AMQ9206E: Error sending data to host ip_address_DP. [ArithInsert1(32), ArithInsert2(32), CommentInsert1(ip_address_DP), CommentInsert2(TCP/IP), CommentInsert3((write))]

    My only guess as to the cause of this error is that the MQ manager is performing a REFRESH SECURITY command on the MQ manager side, which results in the termination of all existing SSL connections. Then I do not understand why DP was the initiator of the connection break.

    As a result of analyzing the situation, I believe that the problem occurred on the DP side.

    At the network level, we have DataPower and MQ Appliance included on the same network equipment. There is no IP Firewall or security equipment for analyzing traffic (like IP Security) between them.

    Since the experience of administration is still little, up to one year, understanding of some already specific situations is still little. Therefore, I hope for your support in this matter.



    ------------------------------
    Andrii Kushneryk
    ------------------------------