MQ

Expand all | Collapse all

MQ behind Load Balancer

  • 1.  MQ behind Load Balancer

    Posted Mon October 05, 2020 04:38 AM
    Hello!

    I've set up MQ, generated certificates, configured channels, queues and so on. Due to my company's security policy, I cant let clients to connect straight to our MQ server. We are using load balancer with wildcard certificate and MQ server sits behind that.

    I have problems with setting up connection with clients and actually for local testing also.

    When I run curl command from linux cli (local machine where MQ is installed) -

    with https, the connection is successful. Server receives "invalid data".

    without https, curls says it gets connected but MQ server doesnt receive any connecton.

    When I configure my "TEST" queue manager (also installed on the same server) with the same CONNAMEs -

    with https (CONNAME('https://my.server.com(1414)')),
    Code:
    AMQ9202E: Remote host not available, retry later.

    RCVR qmgr doesnt receive any connection.

    without https (CONNAME('my.server.com(1414)')), LOGS:
    Code:
    An error occurred receiving data from 'x.x.x.x(1414)' over TCP/IP.  The
    connection to the remote host has unexpectedly terminated.

    RCVR qmgr doesnt receive any connection.


    Why does curl command with HTTPS works, but with channel it doesnt?



    Also, when I test with channel side SSL enabled without Load balancer, everything works. But as Load Balancer requires SSL enabled, I dont see a reason why I need to use SSL also on channels.



    Does anybody have experience with this type of set up? Like I mentioned, I cant let clients connect straight to our MQ Server.


    Any response & information would be useful
    Back to top
    View user's profile Send private message  


    ------------------------------
    Kristjan Voolaid
    ------------------------------


  • 2.  RE: MQ behind Load Balancer

    Posted Mon October 05, 2020 05:18 AM

    It is not completely clear from your description exactly what your setup is. It sounds like you have MQ Clients -> Load Balancer -> Queue Manager. However, you mention testing the connection using curl which would not be able to successfully connect to a queue manager since the queue manager doesn't use HTTP. Are you running the MQ Web Server in front of the queue manager for connecting HTTP clients to perhaps? If not, could you expand on your set up so we can understand the problem properly.

    If you connect an HTTP connection to a queue manager listener, you will get "Invalid Data". What you describe sounds like you get "Invalid data" whether using SSL or not, only first the SSL handshake completes, which is to be expected. There is nothing MQ specific about the SSL Handshake.

    If your MQ Client is using SSL, the MQ Svrconn on the queue manager will also expect SSL.

    How is the load balancer breaking the connection between clients and queue manager - this is not explained either. 

    Hopefully if we can understand your setup better, we can help you solve the problem.

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    ------------------------------



  • 3.  RE: MQ behind Load Balancer

    Posted Mon October 05, 2020 07:04 AM

    Thanks for the response.

    Will try to explain my set up in more detailed way. 

    Currently I have IBM MQ v9.1.5 installed on Linux server

    I have couple of partners who will need to send us messages (MQ server to MQ server.) Due to our security policy, I cant let partners connect straight to our MQ server. So MQ Server sits behind Load balancer, where SSL is enabled and have wildcard certificate. From LB, connection is directed to MQ server.

    I was testing the connection with "TEST QMGR", which is installed to the same instance. I've added our LB DNS to test channel CONNAME but unable to get the connection up.

    FYI, with nc and telnet, I am able to connect to instance through LB. Logs are showing that the connection reached the instance.

    Is it even possible, where client will connect straight from server to our Load balancer, where SSL is enabled?


    Thanks




    ------------------------------
    Kristjan Voolaid
    ------------------------------



  • 4.  RE: MQ behind Load Balancer

    Posted Tue October 06, 2020 12:22 AM

    So this is not MQClient -> Load Balancer -> Queue Manager

    It is Partner Queue Manager -> Load Balancer -> Queue Manager - is that correct?

    One thing to be very careful of here is to ensure that your "Load Balancer" does not do any actual load balancing! The channel from queue manager to queue manager must always go to the correct queue manager.

    You haven't said what this load balancer is, nor how it breaks the connection between your partners and your own queue manager to meet your security requirement. Clearly it is a very important part of the picture. You say when you test without the load balancer it works fine, but with the load balancer in the picture you get "Invalid data". What kind of invalid data? (The FDC will show the data)? Is the load balancer not passing on the connection data exactly as is?

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    ------------------------------



  • 5.  RE: MQ behind Load Balancer

    Posted Tue October 06, 2020 03:43 AM
    Thanks for the answer again.

    Although I found a solution - I needed to add load balancer SSLCIPH to my "TEST" channel. This would activate SSL connection from MQ Server-> Load Balancer. Anyway, thanks for the replays. I am pretty new with IBM so any explanation is very useful!

    ------------------------------
    Kristjan Voolaid
    ------------------------------



  • 6.  RE: MQ behind Load Balancer

    Posted Tue October 06, 2020 05:36 AM
    Hi,

    seems that you got the problem solved. Anyways, here is link that might be useful if you are using LB with IBM MQ:

    https://www.ibm.com/support/pages/some-comments-usage-load-balancers-such-f5-big-ip-mq

    --Hermanni

    ------------------------------
    Hermanni Pernaa
    ------------------------------



  • 7.  RE: MQ behind Load Balancer

    Posted Thu October 22, 2020 03:36 AM
    Yes, I got problem solved with Queue Manager to Queue Manager connection.

    Now I have another partner, who wants to send us messages AS a 'client'. I asked them to add our Load Balancer SslCipher to their connection configuration and partner is able to reach to our MQ Server through Load Balancer  but cant put the "test" message to local queue. Will provide logs from the partner and also from MQ Server.

    Client Logs:

    Code:

    MQHelper.postMessageByJMS  : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'EXAMPLE_QM1' with connection mode 'Client' and host name 'example.com(xxxx)'.
    Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
       at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:595)
       at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
       at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:424)
       at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
       at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7815)
       at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:303)
       at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
       at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6016)
       at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:111)
       at com.thy.tropyaj.pnrgov.MQHelper.postMessageByJMS(MQHelper.java:334)
       at com.thy.tropyaj.pnrgov.SenderManagerThread.sendByMQ(SenderManagerThread.java:147)
       at com.thy.tropyaj.pnrgov.SenderManagerThread.run(SenderManagerThread.java:58)
       at java.lang.Thread.run(Unknown Source)
    Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
       at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)

    Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9204: Connection to host 'texample.com(xxxx)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2009],3=texample.com(xxxx),5=RemoteSession.receiveAsyncTsh]
       at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:2302)
       at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1273)
       at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377)
       at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562)
       at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:357)
       ... 10 more

    Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9213: A communications error for 'TCP' occurred. [1=java.net.SocketException[Socket is closed],4=TCP,5=Socket.setSoTimeout]
       at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.receive(RemoteTCPConnection.java:1800)
       at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.receiveBuffer(RemoteRcvThread.java:733)
       at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.receiveOneTSH(RemoteRcvThread.java:699)
       at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run(RemoteRcvThread.java:139)


    Logs from MQ Server

    Code:


    AMQ9503E: Channel negotiation failed.

    EXPLANATION:
    10/19/2020 10:48:49 AM

    Channel 'xxxxxxxx' between this machine and the remote machine
    'gateway (x.x.x.x)' could not be established due to a negotiation
    failure. In some cases the channel name can not be determined and so is shown
    as '????'.

    The last control data received was type 1 with associated error code
    4294967295.

    ACTION:
    Tell the systems administrator, who should attempt to identify the cause of the
    channel failure using problem determination techniques.  For example, look for
    FFST files, and examine the error logs on the local and remote systems where
    there may be messages explaining the cause of failure.  More information may be
    obtained by repeating the operation with tracing enabled.



    10/19/2020 10:48:49 AM

    AMQ9999E: Channel 'xxxxxxxx' to host 'x.x.x.x' ended
    abnormally.

    EXPLANATION:
    The channel program running under process ID 42348 for channel
    'xxxxxxxxx' ended abnormally. The host name is 'x.x.x.x'; in
    some cases the host name cannot be determined and so is shown as '????'.
    ACTION:
    Look at previous error messages for the channel program in the error logs to
    determine the cause of the failure. Note that this message can be excluded
    completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
    attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
    found in the System Administration Guide.


    Also I see from the Load Balancer logs, that partner is able to reach our MQ Server.


    Does somebody have any clue why this fails?

    Could this be the case that connection from Load Balancer to MQ Server is without SSL?


    Thanks!


    ------------------------------
    Kristjan Voolaid
    ------------------------------



  • 8.  RE: MQ behind Load Balancer

    Posted Thu October 22, 2020 04:05 AM
    We have used the configuration details described in this F5 document: https://www.f5.com/pdf/deployment-guides/ibm-websphere-mq-dg.pdf. Although it is now archived the configuration parameters have worked for us. If I remember correctly we saw similar behavior in the beginning and we were able to solve the issues by adjusting heartbeats/keepalive intervals in all "levels" (firewall, LB, queue managers). Packet capture is a useful tool in cases like these...
    One thing that came to my mind, are you running active-standby cluster of queue managers? If you have tcp_half_open configured on your LB it might allow LB to distribute traffic to the standby node also. Not saying that this is your case, but just something that we noticed in our environment.

    ------------------------------
    Hermanni Pernaa
    ------------------------------



  • 9.  RE: MQ behind Load Balancer

    Posted Thu October 22, 2020 04:12 AM
    And what does the FFST say? Can you add at least the header from the FDC file to this thread?

    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    Website: https://www.mqgem.com
    ------------------------------



  • 10.  RE: MQ behind Load Balancer

    Posted Thu October 22, 2020 08:49 AM
    I am not running any cluster right now. Just one instance.

    FFST doesnt say nothing with.. Atleast with the date when partner was trying to connect.

    ------------------------------
    Kristjan Voolaid
    ------------------------------



  • 11.  RE: MQ behind Load Balancer

    Posted Fri October 23, 2020 02:05 AM
    I'd go with a packet capture. We have usually a network guy running a trace on the firewall or LB 
    and it has proven to be a very efficient way to resolve these kinds of issues.
    After that you'll have some evidence on what is happening.

    --Hermanni

    ------------------------------
    Hermanni Pernaa
    ------------------------------