We run zOSMF on a 6-way sysplex and until recently used automation to start zOSMF (IZUSVR1) on one of the plex members. For various reasons we have decided to remove zOSMF automation from all member systems and use AUTOSTART. Each plex member has:

AUTOSTART(LOCAL)
AUTOSTART_GROUP(IZUDFLT)

specified in IZUPRMxx, and CEA is active on each system. 

Now when we shutdown one of the systems and IPL, we obtain the error:

IZUG038E The file system mounted on user directory is not available.
 M 4000000 GZT1     2023276 08:45:37.05          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
 S                                                         error 138
 E                                           138 00000090  codes were returned: Return code        8, reason code        8

This is despite the fact that there is no IZUSVR1 running on any of the other systems, and we are running a shared-zFS configuration.  If we wait until the IPL is complete, we can manually start IZUSVR1 on the system where it failed, with no issues. It is as if an enqueue on /global/zosmf is in place only during the period of the IPL.

Has anybody else seen something similar ?

Thanks and regards,

Martin

------------------------------
martin hall
------------------------------

Martin,

   In a sysplex, if z/OSMF in each LPAR is configured to be AUTOSTART(LOCAL) and the same AUTOSTART_GROUP(IZUDFLT). In addition, all of z/OSMFs are using the same user directory mounted on a specific zFS filesystem. In this case,  the user directory is /global/zosmf. When the LPARs in the sysplex are IPLed in the meantime, only one z/OSMF is allowed to start. A enqueue with major name 'ZOSMF' and minor name AUTOSTART_GROUP plus zFS file system is used to control the contest. Once the z/OSMF obtains the enq during IPL, other z/OSMFs will failed to start with message IZUG038E appearing in the console. The enqueue obtained by the z/OSMF will be released until the z/OSMF is terminated. So I suppose you saw the message IZUG038E because there was another z/OSMF running on the sysplex which already hold the enqueue not released yet. But you mentioned, after IPL, you were able to manually start the z/OSMF failed to start.  Can you issue the below command from the z/OSMF? I want to check if the AUTOSTART_GROUP is the same as another z/OSMF which starts in IPL. Usually, there must be some configuration such as AUTOSTART_GROUP or server type being different which cause the z/OSMF can be started manually with ignoring the existence of another z/OSMF running.

From both the z/OSMF manually started and the one IPLed, please help issue below commands and take the output for me to investigate.

D GRS,RES=(ZOSMF,*)

Thanks,

We run zOSMF on a 6-way sysplex and until recently used automation to start zOSMF (IZUSVR1) on one of the plex members. For various reasons we have decided to remove zOSMF automation from all member systems and use AUTOSTART. Each plex member has:

AUTOSTART(LOCAL)
AUTOSTART_GROUP(IZUDFLT)

specified in IZUPRMxx, and CEA is active on each system. 

Now when we shutdown one of the systems and IPL, we obtain the error:

IZUG038E The file system mounted on user directory is not available.
 M 4000000 GZT1     2023276 08:45:37.05          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
 S                                                         error 138
 E                                           138 00000090  codes were returned: Return code        8, reason code        8

This is despite the fact that there is no IZUSVR1 running on any of the other systems, and we are running a shared-zFS configuration.  If we wait until the IPL is complete, we can manually start IZUSVR1 on the system where it failed, with no issues. It is as if an enqueue on /global/zosmf is in place only during the period of the IPL.

Has anybody else seen something similar ?

Thanks and regards,

Martin

------------------------------
martin hall
------------------------------

We run zOSMF on a 6-way sysplex and until recently used automation to start zOSMF (IZUSVR1) on one of the plex members. For various reasons we have decided to remove zOSMF automation from all member systems and use AUTOSTART. Each plex member has:

AUTOSTART(LOCAL)
AUTOSTART_GROUP(IZUDFLT)

specified in IZUPRMxx, and CEA is active on each system. 

Now when we shutdown one of the systems and IPL, we obtain the error:

IZUG038E The file system mounted on user directory is not available.
 M 4000000 GZT1     2023276 08:45:37.05          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
 S                                                         error 138
 E                                           138 00000090  codes were returned: Return code        8, reason code        8

This is despite the fact that there is no IZUSVR1 running on any of the other systems, and we are running a shared-zFS configuration.  If we wait until the IPL is complete, we can manually start IZUSVR1 on the system where it failed, with no issues. It is as if an enqueue on /global/zosmf is in place only during the period of the IPL.

Has anybody else seen something similar ?

Thanks and regards,

Martin

------------------------------
martin hall
------------------------------

We run zOSMF on a 6-way sysplex and until recently used automation to start zOSMF (IZUSVR1) on one of the plex members. For various reasons we have decided to remove zOSMF automation from all member systems and use AUTOSTART. Each plex member has:

AUTOSTART(LOCAL)
AUTOSTART_GROUP(IZUDFLT)

specified in IZUPRMxx, and CEA is active on each system. 

Now when we shutdown one of the systems and IPL, we obtain the error:

IZUG038E The file system mounted on user directory is not available.
 M 4000000 GZT1     2023276 08:45:37.05          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
 S                                                         error 138
 E                                           138 00000090  codes were returned: Return code        8, reason code        8

This is despite the fact that there is no IZUSVR1 running on any of the other systems, and we are running a shared-zFS configuration.  If we wait until the IPL is complete, we can manually start IZUSVR1 on the system where it failed, with no issues. It is as if an enqueue on /global/zosmf is in place only during the period of the IPL.

Has anybody else seen something similar ?

Thanks and regards,

Martin

------------------------------
martin hall
------------------------------

Hi Jian,

So  I set up an automatic command to issue D GRS,RES=(ZOSMF,*) on ALL sysplex members at 10-second intervals. I then ensured IZUSVR1 was not running on ANY sysplex members ,before closing down and IPL'ing one of the sysplex members with the following IZU parms :

HOSTNAME('ZOSMFGZP.SPMF.BARCLAYS.CO.UK')
HTTP_SSL_PORT(55444)
SAF_PREFIX('IZUDFLT')
PLUGINS(COMMSERVER_CFG,CAPACITY_PROV,ZERT_ANALYZER,
ISPF,RESOURCE_MON,SOFTWARE_MGMT,SYSPLEX_MGMT,
WORKLOAD_MGMT, INCIDENT_LOG)
SESSION_EXPIRE(15)
JAVA_HOME('/usr/lpp/java/J8.0_64')
KEYRING_NAME('IZUKeyring.IZUDFLT')
AUTOSTART(LOCAL)
AUTOSTART_GROUP('IZUDFLT')
USER_DIR(/global/zosmf/)
CSRF_SWITCH(OFF)

The enqueue command consistently showed:

MR0000000 GZT1     2023286 08:39:26.68 INTERNAL 00000090  IEE421I RO *ALL,D GRS,RES=(ZOSMF 632
LR                                          632 00000090  GZK4     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 326
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZK5     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 307
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT1     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 626
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT2     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 128
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT3     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 954
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT4     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 093
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *

LR                                          632 00000090  GZT5     RESPONSES ---------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 931
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT6     RESPONSES ---------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 074
ER                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *

ie no evidence of an existing enqueue, before the initialization error is issued on the IPL'ing system:

N 4000000 GZT5     2023286 08:42:33.32          00000090  IZUG038E The file system mounted on user directory is not available.
M 4000000 GZT5     2023286 08:42:33.32          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
S                                                         error 804
E                                           804 00000090  codes were returned: Return code        8, reason code        8

Regards,

Martin

We run zOSMF on a 6-way sysplex and until recently used automation to start zOSMF (IZUSVR1) on one of the plex members. For various reasons we have decided to remove zOSMF automation from all member systems and use AUTOSTART. Each plex member has:

AUTOSTART(LOCAL)
AUTOSTART_GROUP(IZUDFLT)

specified in IZUPRMxx, and CEA is active on each system. 

Now when we shutdown one of the systems and IPL, we obtain the error:

IZUG038E The file system mounted on user directory is not available.
 M 4000000 GZT1     2023276 08:45:37.05          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
 S                                                         error 138
 E                                           138 00000090  codes were returned: Return code        8, reason code        8

This is despite the fact that there is no IZUSVR1 running on any of the other systems, and we are running a shared-zFS configuration.  If we wait until the IPL is complete, we can manually start IZUSVR1 on the system where it failed, with no issues. It is as if an enqueue on /global/zosmf is in place only during the period of the IPL.

Has anybody else seen something similar ?

Thanks and regards,

Martin

------------------------------
martin hall
------------------------------

Hi Martin,

   Thanks for the information. From the log, there was no resource related to 'z/OSMF' in any system in the sysplex, but the message IZUG038E  did appear in GZT5. The situation doesn't add up. There is one more thing left to check.  I want to check if there was a z/OSMF started in a system but ending up with failure somehow. In the case, the enqueue was hold by the system but released with the failure. You can find out similar message in the console if existing.

M 0080000 SY1      23289 02:31:37.30          00000000  IRR812I PROFILE IZUSVR1.* (G) IN THE STARTED CLASS WAS USED 443
E                                         443 00000000          TO START IZUSVR1 WITH JOBNAME IZUSVR1.                 

    If the above message is not seen in any system, would you please help open a case on the issue? I need the assistant from GRS team investigating ENQ service.

Thanks,

Hi Jian,

So  I set up an automatic command to issue D GRS,RES=(ZOSMF,*) on ALL sysplex members at 10-second intervals. I then ensured IZUSVR1 was not running on ANY sysplex members ,before closing down and IPL'ing one of the sysplex members with the following IZU parms :

HOSTNAME('ZOSMFGZP.SPMF.BARCLAYS.CO.UK')
HTTP_SSL_PORT(55444)
SAF_PREFIX('IZUDFLT')
PLUGINS(COMMSERVER_CFG,CAPACITY_PROV,ZERT_ANALYZER,
ISPF,RESOURCE_MON,SOFTWARE_MGMT,SYSPLEX_MGMT,
WORKLOAD_MGMT, INCIDENT_LOG)
SESSION_EXPIRE(15)
JAVA_HOME('/usr/lpp/java/J8.0_64')
KEYRING_NAME('IZUKeyring.IZUDFLT')
AUTOSTART(LOCAL)
AUTOSTART_GROUP('IZUDFLT')
USER_DIR(/global/zosmf/)
CSRF_SWITCH(OFF)

The enqueue command consistently showed:

MR0000000 GZT1     2023286 08:39:26.68 INTERNAL 00000090  IEE421I RO *ALL,D GRS,RES=(ZOSMF 632
LR                                          632 00000090  GZK4     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 326
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZK5     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 307
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT1     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 626
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT2     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 128
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT3     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 954
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT4     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 093
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *

LR                                          632 00000090  GZT5     RESPONSES ---------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 931
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT6     RESPONSES ---------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 074
ER                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *

ie no evidence of an existing enqueue, before the initialization error is issued on the IPL'ing system:

N 4000000 GZT5     2023286 08:42:33.32          00000090  IZUG038E The file system mounted on user directory is not available.
M 4000000 GZT5     2023286 08:42:33.32          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
S                                                         error 804
E                                           804 00000090  codes were returned: Return code        8, reason code        8

Regards,

Martin

We run zOSMF on a 6-way sysplex and until recently used automation to start zOSMF (IZUSVR1) on one of the plex members. For various reasons we have decided to remove zOSMF automation from all member systems and use AUTOSTART. Each plex member has:

AUTOSTART(LOCAL)
AUTOSTART_GROUP(IZUDFLT)

specified in IZUPRMxx, and CEA is active on each system. 

Now when we shutdown one of the systems and IPL, we obtain the error:

IZUG038E The file system mounted on user directory is not available.
 M 4000000 GZT1     2023276 08:45:37.05          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
 S                                                         error 138
 E                                           138 00000090  codes were returned: Return code        8, reason code        8

This is despite the fact that there is no IZUSVR1 running on any of the other systems, and we are running a shared-zFS configuration.  If we wait until the IPL is complete, we can manually start IZUSVR1 on the system where it failed, with no issues. It is as if an enqueue on /global/zosmf is in place only during the period of the IPL.

Has anybody else seen something similar ?

Thanks and regards,

Martin

------------------------------
martin hall
------------------------------

Hi Jian

Yes there is definitely something strange going on.

I can see NO evidence of any messages of the kind:

IRR812I PROFILE IZUSVR1.* 

on any of the sysplex members during the period of the shutdown/IPL. A  couple of  further observations:

If while no systems are running zOSMF,  I update parmlib member COMMND00 on ONE  system to specify an automatic start of zOSMF ie:

COM='S IZUANG1'
COM='S IZUSVR1,MSGCLASS=H'

and then shutdown/re-IPL that system, IZUSVR1 starts normally and initializes. 

If I update another system, this time - with no entry in COMMND00 - and to use a different mount point, say /globala/zosmf/, and a different autostart group say IZUDFLTA (and specify LOCAL) , and then shut down and IPL THAT system, I still get the error message at IPL, 

and IZUSVR1 fails to start.

One question - does IZUG038E DEFINITELY relate to the /global/zosmf/ (or equivalent) directory?

If so then there is definitely something odd about the way an AUTOSTARTed IZUSVR1 is trying to access the zFS in our environment. I feel there would be some profit in doing some applicable tracing around the generation of  IZUG038E.

I have raised a case with IBM

Thanks and Regards,

Martin 

Hi Martin,

   Thanks for the information. From the log, there was no resource related to 'z/OSMF' in any system in the sysplex, but the message IZUG038E  did appear in GZT5. The situation doesn't add up. There is one more thing left to check.  I want to check if there was a z/OSMF started in a system but ending up with failure somehow. In the case, the enqueue was hold by the system but released with the failure. You can find out similar message in the console if existing.

M 0080000 SY1      23289 02:31:37.30          00000000  IRR812I PROFILE IZUSVR1.* (G) IN THE STARTED CLASS WAS USED 443
E                                         443 00000000          TO START IZUSVR1 WITH JOBNAME IZUSVR1.                 

    If the above message is not seen in any system, would you please help open a case on the issue? I need the assistant from GRS team investigating ENQ service.

Thanks,

Hi Jian,

So  I set up an automatic command to issue D GRS,RES=(ZOSMF,*) on ALL sysplex members at 10-second intervals. I then ensured IZUSVR1 was not running on ANY sysplex members ,before closing down and IPL'ing one of the sysplex members with the following IZU parms :

HOSTNAME('ZOSMFGZP.SPMF.BARCLAYS.CO.UK')
HTTP_SSL_PORT(55444)
SAF_PREFIX('IZUDFLT')
PLUGINS(COMMSERVER_CFG,CAPACITY_PROV,ZERT_ANALYZER,
ISPF,RESOURCE_MON,SOFTWARE_MGMT,SYSPLEX_MGMT,
WORKLOAD_MGMT, INCIDENT_LOG)
SESSION_EXPIRE(15)
JAVA_HOME('/usr/lpp/java/J8.0_64')
KEYRING_NAME('IZUKeyring.IZUDFLT')
AUTOSTART(LOCAL)
AUTOSTART_GROUP('IZUDFLT')
USER_DIR(/global/zosmf/)
CSRF_SWITCH(OFF)

The enqueue command consistently showed:

MR0000000 GZT1     2023286 08:39:26.68 INTERNAL 00000090  IEE421I RO *ALL,D GRS,RES=(ZOSMF 632
LR                                          632 00000090  GZK4     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 326
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZK5     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 307
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT1     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 626
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT2     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 128
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT3     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 954
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT4     RESPONSES ------------------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 093
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *

LR                                          632 00000090  GZT5     RESPONSES ---------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 931
DR                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *
LR                                          632 00000090  GZT6     RESPONSES ---------------------
DR                                          632 00000090  ISG343I 08.39.25 GRS STATUS 074
ER                                          632 00000090  NO REQUESTORS FOR RESOURCE  ZOSMF    *

ie no evidence of an existing enqueue, before the initialization error is issued on the IPL'ing system:

N 4000000 GZT5     2023286 08:42:33.32          00000090  IZUG038E The file system mounted on user directory is not available.
M 4000000 GZT5     2023286 08:42:33.32          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
S                                                         error 804
E                                           804 00000090  codes were returned: Return code        8, reason code        8

Regards,

Martin

We run zOSMF on a 6-way sysplex and until recently used automation to start zOSMF (IZUSVR1) on one of the plex members. For various reasons we have decided to remove zOSMF automation from all member systems and use AUTOSTART. Each plex member has:

AUTOSTART(LOCAL)
AUTOSTART_GROUP(IZUDFLT)

specified in IZUPRMxx, and CEA is active on each system. 

Now when we shutdown one of the systems and IPL, we obtain the error:

IZUG038E The file system mounted on user directory is not available.
 M 4000000 GZT1     2023276 08:45:37.05          00000090  IZUG002E The address space IZUSVR1 failed to start. The following
 S                                                         error 138
 E                                           138 00000090  codes were returned: Return code        8, reason code        8

This is despite the fact that there is no IZUSVR1 running on any of the other systems, and we are running a shared-zFS configuration.  If we wait until the IPL is complete, we can manually start IZUSVR1 on the system where it failed, with no issues. It is as if an enqueue on /global/zosmf is in place only during the period of the IPL.

Has anybody else seen something similar ?

Thanks and regards,

Martin

------------------------------
martin hall
------------------------------