DS8000 Transparent Cloud Tiering

We have recalled a majority of our data from the Cloud, as we are in the process of rebuilding it.  However the object store still shows there are over 3000 objects in each cluster.  I was wondering when this would appear empty, and how long after that would secondary space management delete the empty containers?  Is there a way to expedite if necessary?

Thanks

John Benik

We have recalled a majority of our data from the Cloud, as we are in the process of rebuilding it.  However the object store still shows there are over 3000 objects in each cluster.  I was wondering when this would appear empty, and how long after that would secondary space management delete the empty containers?  Is there a way to expedite if necessary?

Thanks

John Benik

I also want to mention how TS7700 processes deletes.  The TS7700 will mark the objects as delete-pending internally after the host has sent the delete command.  Afterwrds, if the host comes down and queries for that object/object prefix, we will return 'no records found'.  However, objects may still exist. 

The process of TS7700 deleting the objects is an asynchronous operation.  So, the TS7700 MI will continue to show the counts and space consumption until the objects are physically deleted across the grid.  This process runs once an hour.  Each cluster in the grid will need to assess object copy status and remove the objects. 

Once all instances of an object is deleted, the TS7700 will remove the database entry.  You posted 19 hours ago, I would assume by now all clusters have completed their cleanup process. 

Note:  even if the process of removing the the objects across the grid is still running, you are able to rerun the same job for the same objects.  The TS7700 will treat these delete-pending objects and containers as if they were not there so there is no delay required for cleanup unless you just want to ensure you start with an empty slate for your next test. 

We have recalled a majority of our data from the Cloud, as we are in the process of rebuilding it.  However the object store still shows there are over 3000 objects in each cluster.  I was wondering when this would appear empty, and how long after that would secondary space management delete the empty containers?  Is there a way to expedite if necessary?

Thanks

John Benik

I was just going through the numbers thinking it will go down considerably over the weekend.  

I logged onto all three clusters to look at the totals in each this morning, I fully anticipate this number to drop considerably but have one item I hope you can provide an answer too, I'll discuss at the end.  

In one library in ELR(Elk River) we'll call c0 we have 3238 objects.

In another in ELR we'll call c2 we have 3204 objects.

In the one library in CTC(Chaska)  We have 6494 so 52 more objects then the two Elk River Clusters.

I am expecting that I will see everything but these 52 objects cleaned up, because I suspect these where created when we did a DR test.  Since the host that these where created from is not available is there anyway to clean them up, if we find they are still there come Monday?  My thought was we probably should have done an HDEL on anything we migrated during the DR.

Thanks

John Benik

I also want to mention how TS7700 processes deletes.  The TS7700 will mark the objects as delete-pending internally after the host has sent the delete command.  Afterwrds, if the host comes down and queries for that object/object prefix, we will return 'no records found'.  However, objects may still exist. 

The process of TS7700 deleting the objects is an asynchronous operation.  So, the TS7700 MI will continue to show the counts and space consumption until the objects are physically deleted across the grid.  This process runs once an hour.  Each cluster in the grid will need to assess object copy status and remove the objects. 

Once all instances of an object is deleted, the TS7700 will remove the database entry.  You posted 19 hours ago, I would assume by now all clusters have completed their cleanup process. 

Note:  even if the process of removing the the objects across the grid is still running, you are able to rerun the same job for the same objects.  The TS7700 will treat these delete-pending objects and containers as if they were not there so there is no delay required for cleanup unless you just want to ensure you start with an empty slate for your next test. 

We have recalled a majority of our data from the Cloud, as we are in the process of rebuilding it.  However the object store still shows there are over 3000 objects in each cluster.  I was wondering when this would appear empty, and how long after that would secondary space management delete the empty containers?  Is there a way to expedite if necessary?

Thanks

John Benik

I was just going through the numbers thinking it will go down considerably over the weekend.  

I logged onto all three clusters to look at the totals in each this morning, I fully anticipate this number to drop considerably but have one item I hope you can provide an answer too, I'll discuss at the end.  

In one library in ELR(Elk River) we'll call c0 we have 3238 objects.

In another in ELR we'll call c2 we have 3204 objects.

In the one library in CTC(Chaska)  We have 6494 so 52 more objects then the two Elk River Clusters.

I am expecting that I will see everything but these 52 objects cleaned up, because I suspect these where created when we did a DR test.  Since the host that these where created from is not available is there anyway to clean them up, if we find they are still there come Monday?  My thought was we probably should have done an HDEL on anything we migrated during the DR.

Thanks

John Benik

I also want to mention how TS7700 processes deletes.  The TS7700 will mark the objects as delete-pending internally after the host has sent the delete command.  Afterwrds, if the host comes down and queries for that object/object prefix, we will return 'no records found'.  However, objects may still exist. 

The process of TS7700 deleting the objects is an asynchronous operation.  So, the TS7700 MI will continue to show the counts and space consumption until the objects are physically deleted across the grid.  This process runs once an hour.  Each cluster in the grid will need to assess object copy status and remove the objects. 

Once all instances of an object is deleted, the TS7700 will remove the database entry.  You posted 19 hours ago, I would assume by now all clusters have completed their cleanup process. 

Note:  even if the process of removing the the objects across the grid is still running, you are able to rerun the same job for the same objects.  The TS7700 will treat these delete-pending objects and containers as if they were not there so there is no delay required for cleanup unless you just want to ensure you start with an empty slate for your next test. 

We have recalled a majority of our data from the Cloud, as we are in the process of rebuilding it.  However the object store still shows there are over 3000 objects in each cluster.  I was wondering when this would appear empty, and how long after that would secondary space management delete the empty containers?  Is there a way to expedite if necessary?

Thanks

John Benik

Hi John, 

As Glenn said, we have a DFSMSdss undocumented method of being able to list and Delete the backup objects.   DSS documents the CLOUDUTILS command, information can be found here: https://www.ibm.com/docs/en/zos/2.5.0?topic=commands-cloudutils-command-dfsmsdss

The purpose of CLOUDUTILS is to manage DSS created backups, which means you can list and/or delete backups.  These backups will reside in a DSS named container which has a prefix of 'SYSZADR.'  The undocumented support is to allow a user to have batch CLOUDUTILS perform operations on containers not created by DSS - HSM for example.

Here are the steps you can take:

1. Apply PTFs for APAR OA62674 - This will give you access to the undocumented support for the CONTAINERQUAL keyword of the CLOUDUTILS command.  Specifying CONTAINERQUAL keyword will bypass enforcing DSS to prefix the SYSZADR. to the specified CONTAINER and instead keep the specified CONTAINER as-is.
2. The use of the CONTAINERQUAL keyword is RACF protected. The user must have read access to the facility class profile specifying the container that corresponds to the container being LISTed or DELETEd.  Without giving the proper RACF authority, you will receive error ADR707E.  You must define a facility class modeled like:  STGADMIN.ADR.CLOUD.'qual'. Where 'qual' is a portion of the specified CONTAINER.  This is simply a method to protect non-DFSMSdss resources.
   1. Example:  An HSM backup resides in the TS7700 in a container named SYSZARC.ARCTST.MIG.2023001.5BD77F00 and another named SYSZARC.ARCTST.DMP.2023002.5BD77E7A. You want to cleanup MIG resources.  Your RACF profile would look like: 'STGADMIN.ADR.CLOUD.SYSZARC.ARCTST.MIG'
3. Use DFSMSdss Batch JCL to invoke CLOUDUTILS command:
   1. If you do not know the OBJECTPREFIX that HSM used, you can list all resources.  Otherwise, skip to step 3.4. Example:  For the container we identified in step 2.1 above, list the resources:

   2. CLOUDUTILS LIST CONTAINER(SYSZARC.ARCTST.MIG.2023001.5BD77F00) - 
      CONTAINERQUAL(3) CLOUD(mycloudname) CCREDS(mycredentials)

   3. Update with cloud name and credentials you provided to HSM.   Notice that CONTAINERQUAL specifies 3, because it's the first 3 qualifiers of the CONTAINER that were RACF indicated.  Note:  There is a limit of 20 characters for the 'qual' portion.
   4. When ready, Delete the object resources comprising of the HSM backup as identified with the OBJECTPREFIX:

   5. CLOUDUTILS DELETE CONTAINER(SYSZARC.ARCTST.MIG.2023001.5BD77F00) -     
       OBJECTPREFIX(myobjectprefix) -                                                         
       CONTAINERQUAL(3) - 
       CLOUD(mycloudname) CCREDS(my credentials)

4. Refer to the CLOUDUTILS documentation for further explanation of how to use CLOUDUTILS.
5. IF you have issues using the undocumented support.  Please feel free to ask follow on questions either here or at my email: erfiguer@us.ibm.com

I was just going through the numbers thinking it will go down considerably over the weekend.  

I logged onto all three clusters to look at the totals in each this morning, I fully anticipate this number to drop considerably but have one item I hope you can provide an answer too, I'll discuss at the end.  

In one library in ELR(Elk River) we'll call c0 we have 3238 objects.

In another in ELR we'll call c2 we have 3204 objects.

In the one library in CTC(Chaska)  We have 6494 so 52 more objects then the two Elk River Clusters.

I am expecting that I will see everything but these 52 objects cleaned up, because I suspect these where created when we did a DR test.  Since the host that these where created from is not available is there anyway to clean them up, if we find they are still there come Monday?  My thought was we probably should have done an HDEL on anything we migrated during the DR.

Thanks

John Benik

I also want to mention how TS7700 processes deletes.  The TS7700 will mark the objects as delete-pending internally after the host has sent the delete command.  Afterwrds, if the host comes down and queries for that object/object prefix, we will return 'no records found'.  However, objects may still exist. 

The process of TS7700 deleting the objects is an asynchronous operation.  So, the TS7700 MI will continue to show the counts and space consumption until the objects are physically deleted across the grid.  This process runs once an hour.  Each cluster in the grid will need to assess object copy status and remove the objects. 

Once all instances of an object is deleted, the TS7700 will remove the database entry.  You posted 19 hours ago, I would assume by now all clusters have completed their cleanup process. 

Note:  even if the process of removing the the objects across the grid is still running, you are able to rerun the same job for the same objects.  The TS7700 will treat these delete-pending objects and containers as if they were not there so there is no delay required for cleanup unless you just want to ensure you start with an empty slate for your next test. 

We have recalled a majority of our data from the Cloud, as we are in the process of rebuilding it.  However the object store still shows there are over 3000 objects in each cluster.  I was wondering when this would appear empty, and how long after that would secondary space management delete the empty containers?  Is there a way to expedite if necessary?

Thanks

John Benik

Hi John, 

As Glenn said, we have a DFSMSdss undocumented method of being able to list and Delete the backup objects.   DSS documents the CLOUDUTILS command, information can be found here: https://www.ibm.com/docs/en/zos/2.5.0?topic=commands-cloudutils-command-dfsmsdss

The purpose of CLOUDUTILS is to manage DSS created backups, which means you can list and/or delete backups.  These backups will reside in a DSS named container which has a prefix of 'SYSZADR.'  The undocumented support is to allow a user to have batch CLOUDUTILS perform operations on containers not created by DSS - HSM for example.

Here are the steps you can take:

1. Apply PTFs for APAR OA62674 - This will give you access to the undocumented support for the CONTAINERQUAL keyword of the CLOUDUTILS command.  Specifying CONTAINERQUAL keyword will bypass enforcing DSS to prefix the SYSZADR. to the specified CONTAINER and instead keep the specified CONTAINER as-is.
2. The use of the CONTAINERQUAL keyword is RACF protected. The user must have read access to the facility class profile specifying the container that corresponds to the container being LISTed or DELETEd.  Without giving the proper RACF authority, you will receive error ADR707E.  You must define a facility class modeled like:  STGADMIN.ADR.CLOUD.'qual'. Where 'qual' is a portion of the specified CONTAINER.  This is simply a method to protect non-DFSMSdss resources.
   1. Example:  An HSM backup resides in the TS7700 in a container named SYSZARC.ARCTST.MIG.2023001.5BD77F00 and another named SYSZARC.ARCTST.DMP.2023002.5BD77E7A. You want to cleanup MIG resources.  Your RACF profile would look like: 'STGADMIN.ADR.CLOUD.SYSZARC.ARCTST.MIG'
3. Use DFSMSdss Batch JCL to invoke CLOUDUTILS command:
   1. If you do not know the OBJECTPREFIX that HSM used, you can list all resources.  Otherwise, skip to step 3.4. Example:  For the container we identified in step 2.1 above, list the resources:

   2. CLOUDUTILS LIST CONTAINER(SYSZARC.ARCTST.MIG.2023001.5BD77F00) - CONTAINERQUAL(3) CLOUD(mycloudname) CCREDS(mycredentials)

   3. Update with cloud name and credentials you provided to HSM.   Notice that CONTAINERQUAL specifies 3, because it's the first 3 qualifiers of the CONTAINER that were RACF indicated.  Note:  There is a limit of 20 characters for the 'qual' portion.
   4. When ready, Delete the object resources comprising of the HSM backup as identified with the OBJECTPREFIX:

   5. CLOUDUTILS DELETE CONTAINER(SYSZARC.ARCTST.MIG.2023001.5BD77F00) -      OBJECTPREFIX(myobjectprefix) -                                                          CONTAINERQUAL(3) -  CLOUD(mycloudname) CCREDS(my credentials)

4. Refer to the CLOUDUTILS documentation for further explanation of how to use CLOUDUTILS.
5. IF you have issues using the undocumented support.  Please feel free to ask follow on questions either here or at my email: erfiguer@us.ibm.com

I was just going through the numbers thinking it will go down considerably over the weekend.  

I logged onto all three clusters to look at the totals in each this morning, I fully anticipate this number to drop considerably but have one item I hope you can provide an answer too, I'll discuss at the end.  

In one library in ELR(Elk River) we'll call c0 we have 3238 objects.

In another in ELR we'll call c2 we have 3204 objects.

In the one library in CTC(Chaska)  We have 6494 so 52 more objects then the two Elk River Clusters.

I am expecting that I will see everything but these 52 objects cleaned up, because I suspect these where created when we did a DR test.  Since the host that these where created from is not available is there anyway to clean them up, if we find they are still there come Monday?  My thought was we probably should have done an HDEL on anything we migrated during the DR.

Thanks

John Benik

I also want to mention how TS7700 processes deletes.  The TS7700 will mark the objects as delete-pending internally after the host has sent the delete command.  Afterwrds, if the host comes down and queries for that object/object prefix, we will return 'no records found'.  However, objects may still exist. 

The process of TS7700 deleting the objects is an asynchronous operation.  So, the TS7700 MI will continue to show the counts and space consumption until the objects are physically deleted across the grid.  This process runs once an hour.  Each cluster in the grid will need to assess object copy status and remove the objects. 

Once all instances of an object is deleted, the TS7700 will remove the database entry.  You posted 19 hours ago, I would assume by now all clusters have completed their cleanup process. 

Note:  even if the process of removing the the objects across the grid is still running, you are able to rerun the same job for the same objects.  The TS7700 will treat these delete-pending objects and containers as if they were not there so there is no delay required for cleanup unless you just want to ensure you start with an empty slate for your next test. 

We have recalled a majority of our data from the Cloud, as we are in the process of rebuilding it.  However the object store still shows there are over 3000 objects in each cluster.  I was wondering when this would appear empty, and how long after that would secondary space management delete the empty containers?  Is there a way to expedite if necessary?

Thanks

John Benik