MQ

 View Only
  • 1.  Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    Posted 14 days ago

    Hello All,

    Recently, we tried to create a distribution setup between our old on-premise Linux server and a newly created AWS qmgr on Redhat OpenShift platform(Both internal). We basically created the setup and configured the channels(sender and receiver) with SSLV1.2(SHA256) and trying to establish communication.But,we are not able to do so.

    So,whenever i try to ping the channel and establish communication ,i am getting "AMQ9665: SSL connection closed by remote end of channel ''. (We have checked all the things like firewall between servers,all the loadbalancer working properly etc).Everything seems fine.But,we are not able to make a successful connection even after adding the SSL certificates we received from CA(internal to the company),exchanged the public to other side qmgr and "Refreshed the SSL Security".

    But using the same certs,we are able to connect to the AWS qmgr and PUT/GET messages from local machine as a client communication.But,not able to do server-server communication.

    Note : Here our AWS is configured in such a way that any external connection to AWS MQ should have to go through the load balancer(with SSL configuration) which then redirects to 3 available IP-Addresses at MQ level.At Load balancer level,any incoming connection with SSL can only pass through and others(TCP etc) will be rejected.

    We tried several things from our end,like updating the same SSLCIPH value(TLSV1.2 or V1.3) at both ends of channels,SSLPEER(CN=* or CN=linux side CN name etc),updating the same CERTLABL at both ends of the qmgr and such things.But always we get the same "SSL connection closed by remote end of the channel" and the channel goes into retrying state and the error logs state as below.

    -------------------------------------------------------------------------------
    06/20/2024 06:16:28 PM - Process(2496.1) User(mqm) Program(runmqchl)
                        Host(lxapp4563) Installation(Installation1)
                        VRMF(8.0.0.7) QMgr(TEST)
     
    AMQ9665: SSL connection closed by remote end of channel 'MQHPSA20.TO.EXAMPQM'.
     
    EXPLANATION:
    The SSL or TLS connection was closed by the remote host '10.231.24.161(443)'
    during the secure socket handshake. The channel is 'MQHPSA20.TO.EXAMPQM'; in
    some cases its name cannot be determined and so is shown as '????'. The channel
    did not start.
    ACTION:
    Check the remote end of the channel for SSL and TLS errors. Fix them and
    restart the channel.
    ----- amqccisa.c : 7839 -------------------------------------------------------
    06/20/2024 06:16:28 PM - Process(2496.1) User(mqm) Program(runmqchl)
                        Host(lxapp4563) Installation(Installation1)
                        VRMF(8.0.0.7) QMgr(TEST)
     
    AMQ9999: Channel 'MQHPSA20.TO.EXAMPQM' to host
    'x.apps.np01.moss.corp.com(443)' ended abnormally.
     
    EXPLANATION:
    The channel program running under process ID 2496 for channel
    'MQHPSA20.TO.EXAMPQM' ended abnormally. The host name is
    'x.apps.np01.moss.corp.com(443)'; in some cases the host name cannot be
    determined and so is shown as '????'.
    ACTION:
    Look at previous error messages for the channel program in the error logs to
    determine the cause of the failure. Note that this message can be excluded
    completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
    attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
    found in the System Administration Guide.

    So,can anyone suggest what can be done for this to get Resolved and make channels come into running state.

    Note : Also,we tried to update some properties such "OUTBOUNDSNI" value at Linux side 'qm.ini' and check once qmgr is restarted.Then we re able to make a successful 'PING' to remote end(AWS),but was not able to start the channel.

    Below are the errors at that time.

    So,any idea what can be done to resolve this issue?



    ------------------------------
    Avinash Kaja
    ------------------------------


  • 2.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    IBM Champion
    Posted 13 days ago

    Hi Avinash,

    The error AMQ9665 that you have shown us says the following in the ACTIION section:-

    Check the remote end of the channel for SSL and TLS errors. Fix them and restart the channel.

    Have you reviewed the error log at the other end of the channel? Can you show us the errors that are reported there?

    wrt your ini file error, you will have to show us what value you put in there in order for us to help you.

    I can see that you have spelled OutboundSNI correctly, so that means it must be complaining about the value. What value did you set it to? Allowed values are the keywords CHANNEL or HOSTNAME.

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    Website: https://www.mqgem.com
    ------------------------------



  • 3.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    IBM Champion
    Posted 13 days ago

    Oops! Just spotted that you are using MQ version 8 - so therefor OutboundSNI is not supported yet. You'll need instead to use the now deprecated AllowOutboundSNI ini file attribute instead. Have a read of SSL stanza of the client configuration file to learn more about that.

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    Website: https://www.mqgem.com
    ------------------------------



  • 4.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    IBM Champion
    Posted 13 days ago

    Actually - no strike that, even AllowOutboundSNI is not available until 9.1!

    V8 is really rather old - is there an opportunity for you to upgrade to something a little bit more up-to-date? You just don't seem to have the features you need in that well out of date release.

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    Website: https://www.mqgem.com
    ------------------------------



  • 5.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    Posted 11 days ago

    Hello Hughson,

    Thanks for your response.

    We don't have the approval as of now from client about upgrading the version 8 of ours to latest in our on-premise linux servers.We wanted to try the distributed setup between this existing environment itself.

     Also, coming to the below question about AMQ9665 when we get the channel retry error and ping issue and see SSL/TLS issues in sender side,we are not seeing any errors getting recorded at remote end.

    "The error AMQ9665 that you have shown us says the following in the ACTIION section:-

    Check the remote end of the channel for SSL and TLS errors. Fix them and restart the channel."

    Could you let me know if there is a way we try successfully establish the connection between this on-premise Linux and MQ on RedHat Openshift MQ with SSL in place,not as a client but server-server communication?

    FYI,

    On-premise Linux - MQ Version 8

    ROSAWS MQ : Version 9.3.5.

    Are there any attributes or SSL stanza's that need to be updated?

    We Have uploaded SSL certs in both ends(On-Premise Main cert at AWS MQ side and Vice-Versa),Updated the SSLCIPH value to 'TLS_RSA_WITH_AES_256_CBC_SHA256',SSLPEER at AWS qmgr is set to 'CN=*' and SSLCAUTH attribute is set to 'Optional'.    Still,couldn't make a successful connection.Is there any reason that you might know for this? If So,Could you let me know the solution.



    ------------------------------
    Avinash Kaja
    ------------------------------



  • 6.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    Posted 11 days ago
    Edited by Diego Mendoza 11 days ago

    Hi Avinash,

    Unfortunately, it is not possible to establish a successful connection between an on-premise Queue Manager in MQ Version 8 and MQ on RedHat OpenShift MQ using SSL for server-to-server communication. I recommend following Morag Hughson's advice to upgrade the MQ version of the on-premise Queue Manager.

     A few months ago, I was in a similar situation to yours. I needed to interconnect a Queue Manager deployed on-premise with a Queue Manager deployed in ROSA under a Native HA configuration.

    In my case, the only way to get the configuration to work was by updating the Queue Manager to a recent version. Starting with IBM MQ version 9.2.1, the OutboundSNI property can be configured in the qm.ini file. This variable can be set to either CHANNEL (default) or HOSTNAME, indicating whether the certificate configured at the Queue Manager level or at the Channel level should be presented.

    Reference: Configuring a Route to connect to a queue manager from outside a Red Hat OpenShift cluster


    ------------------------------
    Diego Mendoza
    Cloud Architect
    MicroGestion Software
    Argentina
    DIEGO_MENDOZA@MICROGESTION.COM
    ------------------------------



  • 7.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    Posted 11 days ago

    Thanks Diego/Hughson for your response.

    So, basically what you are saying is the version difference between On-Premise and AWS ROSA qmgr is what is stopping the SSL connection to come out as successful and Without this upgradation, we won't be able to successfully establish a connection, right?

    Also,a quick doubt about the OutboundSNI property,this is generally a property for client channel itself,right?Is that something we can use for a server-server communication?



    ------------------------------
    Avinash Kaja
    ------------------------------



  • 8.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    Posted 11 days ago
    Edited by Diego Mendoza 11 days ago

    Hi Avinash,

    Red Hat OpenShift Route depends on the Server Name Indication (SNI) behavior, so you need two IBM MQ installations with SNI support on both sides of the connection. It doesn't matter if different versions of IBM MQ are used, as long as both versions support SNI. 

    In short, TLS does not provide a mechanism for a client to tell a server the name of the server it is contacting. SNI is an extension to TLS that allows a server receiving a client hello containing the "server_name" extension to guide its selection of an appropriate certificate to return to the client. 

    If in an IBM MQ connection the SNI-related header is not present when the on-premise side tries to connect to the queue manager deployed in ROSA, OpenShift will present its own certificate, causing the connection to fail.

    OutboundSNI is not a channel property; it's more like an environment variable. For example:

    • For interconnecting Queue Managers, the OutboundSNI property in the SSL stanza of the qm.ini file (on the on-premise side) must be set.

    • For IBM MQ client connections from outside the OpenShift cluster, the OutboundSNI property in the SSL stanza of the IBM MQ client configuration file must be set.

    • For a Java application using MQ Classes connecting from outside OpenShift, the com.ibm.mq.cfg.SSL.outboundSNI environment variable must be set.



    ------------------------------
    Diego Mendoza
    Cloud Solutions Architect
    MicroGestion Software
    Argentina
    DIEGO_MENDOZA@MICROGESTION.COM
    ------------------------------



  • 9.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    IBM Champion
    Posted 10 days ago

    Hello,

    May be you can have a look at MQ IPT (MQ Internet Passrhru).

    It act as a relay between two MQ "devices", you can have differents cipher on both sides, ...

    I used it recently to connect a MQ 7.1 Queue Manager to a 9.2 one, using TLS 1.2.

    Hope this help, LMD. 



    ------------------------------
    Luc-Michel Demey
    DEMEY CONSULTING
    lmd@demey-consulting.fr
    #IBMChampion
    ------------------------------



  • 10.  RE: Issue with distribution setup between on-premise Linux and AWS on ROSA(REDHATOPENSHIFT) Platform

    Posted 9 days ago

    Hi Avinash,
    There is a workaround that can be used to bypass the Server Name Indication (SNI) used by OpenShift/EKS. I have used it with AWS EKS.
    You can route your traffic directly to your Qmgr port 1414 with a AWS Network LoadBalancer (NLB)
    This allows connecting to your Qmgr with or without SSL/TLS. If you have more than one Qmgr, you will need one NLB per Qmgr.
    It can be achieved with a Service type: LoadBalancer like this sample based on the MQ Native HA Helm Chart.

    _________________________________
    apiVersion: v1
    kind: Service
    metadata:
      annotations:
        meta.helm.sh/release-name: qm1
        meta.helm.sh/release-namespace: default
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
      labels:
        app.kubernetes.io/instance: qm1
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: ibm-mq
        app.kubernetes.io/version: 9.3.0.0
        helm.sh/chart: ibm-mq-3.0.0
      name: qm1-ibm-mq-loadbalancer
      namespace: default
    spec:
      externalTrafficPolicy: Local
      type: LoadBalancer
      ports:
      - port: 1414
        name: qmgr
      selector:
        app.kubernetes.io/instance: qm1
        app.kubernetes.io/name: ibm-mq
    _________________________________

    Hope it can help



    ------------------------------
    martin rajotte
    ------------------------------