Cluster sender channels, are the right ones defined?

David Ware |Aug 7 2014 Updated

Recent discussions I’ve had have made me realise that there are many MQ clusters out there that have cluster sender channels manually defined where they shouldn’t be, and possibly some missing where they really should be. This could be down to remnants of previous configurations or simple misunderstandings of what’s needed. So I thought I’d try to clarify things a little, there are basically two simple rules, so this should be quick…

1. A full repositoryqueue manager must have a manually defined cluster sender channel pointing to every other queue manager that is also a full repository for each of the clusters that this queue manager is a full repository for.
2. For each cluster that a queue manager is only apartial repository for there should only be one manually defined cluster sender channel pointing at one of the queue managers that hosts a full repository for that cluster.

A little more info on these…

1. Full repositories
2. A full repository queue manager must have a manually defined cluster sender channel pointing to every other queue manager that is also a full repository for each of the clusters that this queue manager is a full repository for.

This is necessary so that cluster information learnt by one full repository is replicated across the other full repositories. Obviously, there will normally only be one other full repository (see here), but occasionally you may have a third or forth during a full repository migration phase, in which case you must have manually defined cluster sender channels between each of them.

2. Partial repositories

For each cluster that a  queue manager is only a partial repository for there should only be one manually defined cluster sender channel pointing at one of the queue managers that hosts a full repositories for that cluster.

Many people manually define a cluster sender channel to more than one full repository queue manager, either because they believe this is necessary or because they think it will dictate which full repositories are to be used. Neither is really true…

Once a partial repository queue manager has communicated with a full repository (over the first manual cluster sender channel it detects) it will be told by that full repository of a second full repository to use for HA purposes, this is then remembered over queue manager restarts. So it doesn’t need an explicit channel definition to a second full repository, the first full repository sorts that all out automatically.

And despite it appearing to behave in this way at first glance, if you do have more than two full repositories and you do manually define that second channel it doesn’t 100% guarantee that the two full repositories that this queue manager uses will be the two pointed to by the two manual cluster sender definitions. There are edge conditions where this is not the case as this was never the intended use. So to save future confusion, just keep it simple and have the single defined cluster sender channel.

In both of the above cases, if multiple clusters are involved with the same full repository queue managers it doesn’t matter if there is a single cluster sender channel definition using a cluster namelist to list all the clusters or individual cluster sender channel definitions for each cluster, the same rules apply.

It’s also worth pointing out that it’s possible for a queue manager to be a full repository for some clusters and a partial repository for others, that’s why the above rules are described in terms of clusters rather than queue managers as a whole.

So, is your setup correct?

If you’re responsible for maintaining the queue managers in a cluster, especially one that’s grown over time, it can be hard to work out from channel configuration alone what is right and what is wrong, especially if there are multiple overlapping clusters to consider.

At the time MQ detects an invalid configuration it will report it to the queue manager error logs, where you’ll see this message (CSQX427E on z/OS):

AMQ9427: CLUSSDR channel does not point to a repository queue manager.

But it’s easy to miss those, so how do you check when the system’s configured? Obviously you can check all the channel and repository configuration on each queue manager and work it out but another way to check for a correct configuration that doesn’t need you to cross reference between multiple queue manager’s definitions is to check each queue manager’s cluster queue manager entries through the MQSC command DISPLAY CLUSQMGR (this also applies when using MQExplorer but I’ll concentrate on MQSC here).

Each CLUSQMGR entry represents how the local queue manager sees the other queue managers in each of the clusters it is a member of. You get a separate entry per cluster irrespective of whether namelists have been used on the cluster channels or not. The entry contains two particularly useful attributes for this discussion, QMTYPE and DEFTYPE.

QMTYPE simply shows if the other queue manager is a full repository (‘REPOS’) for the cluster or a partial repository (‘NORMAL’).

DEFTYPE shows you how the relationship between the queue managers has been established, based on what cluster channels have been defined.

DEFTYPE has a number of rather cryptic values, CLUSSDR, CLUSSDRA, CLUSSDRB and CLUSRCVR. We’re working on improving the MQ documentation for these, but in the meantime, I’ll summarise them here, so a short diversion…

DEFTYPE values

CLUSRCVR: This is the entry for the local queue manager in each cluster it has a cluster receiver channel defined.

Any CLUSSDR* value means this entry represents a remote queue manager in a cluster. The different values however help you understand how the local queue manager came to know about it:

CLUSSDRA: This is a remote cluster queue manager that the local queue manager has no manually defined cluster sender channel for but has been told about it by someone else, either by the remote queue manager itself (typically because this queue manager is a full repository) or because a full repository has told this queue manager about it as it needs to communicate with it for some reason.

CLUSSDRB: This means the local queue manager has a manually defined cluster sender channel which has been used to establish contact with the target queue manager and that queue manager has accepted it from the point of view of the cluster. The target could be a full or a partial repository, although as I’ve already said you really only want it to be a full repository at the other end.

CLUSSDR: This means the local queue manager has manually defined a cluster sender channel to the remote queue manager but the initial cluster handshake between them has not yet completed. This may be because the channel has never started, perhaps because the target is not running or the configuration is incorrect. It could also mean the channel has successfully started but the target queue manager did not like the cluster information provided, for example a cluster name was set in the cluster sender channel definition that does not match the target’s cluster membership. Once the handshake has been performed the DEFTYPE should change to CLUSSDRB, so in a healthy system CLUSSDR should only be a transitory state.

Anyway, where were we? Oh yes, working out if our cluster channel configuration is correct or not. The simplest way I’ve found is to check that any CLUSSDRB queue managers are pointing at a full repository and that if the queue manager is only a partial repository, it only has the one CLUSSDRB per cluster. So issuing the following on a queue manager allows you to see that…

DISPLAY CLUSQMGR(*) WHERE(DEFTYPE EQ CLUSSDRB) QMTYPE

Then check that all the QMTYPEs in the results are set to REPOS. Any that say NORMAL indicate a manual cluster sender pointing to a partial repository which is incorrect and needs some investigation.

You now check which clusters the local queue manager is a full repository for, you can use the CLUSQMGR entries again to do that by looking at the QMTYPE for each entry with a DEFTYPE of CLUSRCVR…

DISPLAY CLUSQMGR(*) WHERE(DEFTYPE EQ CLUSRCVR) QMTYPE

(Alternatively you can look at the REPOS and REPOSNL attributes of the queue manager)

With this full repository information crosscheck the CLUSQMGR results from above to make sure that for every cluster that this queue manager is not a full repository for there is only a single CLUSQMGR returned for that cluster with a DEFTYPE of CLUSSDRB.

If this queue manger is a full repository for any clusters you now need to issue the following command…

DISPLAY CLUSQMGR(*) WHERE(QMTYPE EQ REPOS) DEFTYPE

Now, for each entry that matches a cluster that this queue manager is a full repository for you need to make sure the DEFTYPE is set to CLUSSDRB and not CLUSSDRA (ignoring the local, CLUSRCVR entries). If there are any CLUSSDRAs you need to manually define a cluster sender channel to that queue manager.

Once you’ve done this you should have the right cluster sender channel configuration for that queue manager.

But before I go, if you issue the following command and get any results back it means there are cluster sender channels defined that have not been correctly established so you should look into why that’s the case (check the configuration, the availability of the other end and for any error messages written by either end mentioning that channel/queue manager)…

DISPLAY CLUSQMGR(*) WHERE(DEFTYPE EQ CLUSSDR)

Logically thinking…

For those of you that likes to think in pseudo code, or want to automate the checking, try this….

First build a list of all clusters that the local queue manager is a full repository (FR) for.

<e.g. DISPLAY CLUSQMGR(*) WHERE(DEFTYPE EQ CLUSRCVR)>

For each CLUSQMGR with a DEFTYPE of CLUSSDR*:

<e.g. DISPLAY CLUSQMGR(*) WHERE(DEFTYPE NE CLUSRCVR)>

      If DEFTYPE=CLUSSDRB:

                  Is QMTYPE=NORMAL?

Error: You shouldn’t have a manual cluster sender to a partial repository.

                  Is the local queue manager a partial repository for this cluster?

                              Have we already seen a CLUSSDRB for this cluster?

                                          Error: You should only have one manual cluster sender defined.

      If DEFTYPE=CLUSSDRA:

                  Is QMTYPE=REPOS and the local queue manager is a FR for this cluster?

                              Error: All FR to FR channels should be manually defined.

      If DEFTYPE=CLUSSDR:

                  Error: The cluster handshake has not completed.

And finally...

If you do want to investigate this with MQExplorer you need to look at the “Cluster-sender Channels” tab for each queue manager listed under the “Queue Manager Clusters” folders. Then check the “Queue manager type” (i.e. QMTYPE) and “Definition type” (i.e. DEFTYPE) for each of them.

That was longer than I was expecting, so thanks for reading to the end, I hope it helps to clarify things.