z/OS DFSMS

z/OS DFSMS

z/OS DFSMS

z/OS Data Subsystem that delivers core Data, Data Resiliency and Data Lifecycle Management Solutions for your Enterprise.

 View Only
Expand all | Collapse all

Dfdss dump to minio S3 cloud?

  • 1.  Dfdss dump to minio S3 cloud?

    Posted Fri June 21, 2024 10:00 AM

    Hi,

    I currently use dfdss for daily off-site ftp backups.

    Currently I do the following:

    • Use dfdss to dump the required datasets to dasd, then
    • Use TSO transmit to create a portable version of the dump and finally,
    • Copy the xmit file to an ftp server using binary ftp

    The above steps work & allow dataset restoration but takes consideration space & CPU time.

    Instead of all the above, I was wondering if z/OS 3.1 dfdss allows directly uploading/ restoring a logical dataset dump to & from a minio or other S3 cloud?

    Thanks,

    Travis 



    ------------------------------
    Travis Thorne
    ------------------------------


  • 2.  RE: Dfdss dump to minio S3 cloud?

    Posted Fri June 21, 2024 12:15 PM

    Hi Travis, 

    Do you have DS8900 storage?  DFSMSdss has the ability of creating your backups on an S3-compatible cloud or even TS7700 object storage using DS8K  (Transparent Cloud Tiering) TCT.  

    Other clients have also attempted what you are wanting using CDA...there is a blogpost about it here: https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/john-hogan/2024/05/06/using-gdkutil-to-copy-backup-tape-data-to-an-s3-co?communityKey=e7b7d299-8509-4572-8cf1-c1112684644f

    I would be happy to have a call with you and talk about how to use TCT.  If you do not have DS8K storage or do not want to use TCT, we can still have a call and discuss roadmap items under an  NDA.

    Thanks, 

    Ernesto E. Figueroa



    ------------------------------
    ERNESTO FIGUEROA
    ------------------------------



  • 3.  RE: Dfdss dump to minio S3 cloud?

    Posted Fri July 12, 2024 12:18 AM

    Hi Ernesto,

    Thanks for your reply, sorry I just found it as I just realised outlook classified everything from IBM Communities as spam :(

    For our production data, we use Dell Powermax 8000s; which from what I understand can't yet be used with TCT.

    It'd be really great if DFDSS could upload directly to S3 storage  - should I create an IBM Idea?

     

    I've had a play with GDKUtil & got it to upload / download small files to a local minio server.

    I'll continue to pursue that option, as it looks like it might work with larger DFDSS dumps.

    For example this is one of the dumps I'll try and upload to minio using gdkutil soon:

                                 Data Set Information                         
    Command ===>                                                              
                                                                              
    Data Set Name  . . . : OFFSITE.????.DAILY.G0313V00                       
                                                                              
    General Data                          Current Allocation                  
     Management class . . : PRIMARY        Allocated cylinders : 13,216
     Storage class  . . . : SCBACKUP       Allocated extents . : 4
      Volume serial . . . : ?$BU02 +                                          
      Device type . . . . : 3390                                              
     Data class . . . . . : DCBACKUP                                          
      Organization  . . . : PS            Current Utilization                 
      Record format . . . : U              Used cylinders  . . : 13,216
      Record length . . . : 0              Used extents  . . . : 4
      Block size  . . . . : 32760                                             
      1st extent cylinders: 1000                                              
      Secondary cylinders : 5000          Dates                               
      Data set name type  : LARGE          Creation date . . . : 2024/07/11   
      Data set encryption : NO             Referenced date . . : 2024/07/11   
                                           Expiration date . . : ***None***   
                                                                              
      SMS Compressible  . : NO                                                
                                                                              

    Thanks,

    Travis



    ------------------------------
    Travis Thorne
    ------------------------------



  • 4.  RE: Dfdss dump to minio S3 cloud?

    Posted Fri July 12, 2024 11:04 AM

    No worries, I am glad you were able to setup and play with GDKUTILS.

    Please feel free to contact me (erfiguer@us.ibm.com) if you would like to discuss more!



    ------------------------------
    ERNESTO FIGUEROA
    ------------------------------



  • 5.  RE: Dfdss dump to minio S3 cloud?

    Posted Mon February 17, 2025 09:50 AM

    Hi Travis, I wanted to follow up with you on this topic.  

    Please look at our DFSMSdss enhancement OA66450 shipped in 4Q25.    This is our direct-to-cloud support which leverages the latest DFSMSdfp Cloud Data Access (CDA) access method.

    With this enhancement you can accomplish specifically what you are looking for in a single step for Dump.

    I hope this helps!



    ------------------------------
    ERNESTO FIGUEROA DFSMSdss Product Owner
    ------------------------------



  • 6.  RE: Dfdss dump to minio S3 cloud?

    Posted Mon February 17, 2025 07:28 PM

    Hi Ernesto,

    Thanks for following up on this ,

    I'll build get the ptf asap & try this out.

    Kind regards,

    Travis Thorne



    ------------------------------
    Travis Thorne
    ------------------------------



  • 7.  RE: Dfdss dump to minio S3 cloud?

    Posted Tue February 18, 2025 12:26 PM

    Correction to my post, I incorrectly said 4Q25....This was meant to be 4Q24!  My apologies, this support is available today:

    https://www.ibm.com/support/pages/apar/OA66450



    ------------------------------
    ERNESTO FIGUEROA DFSMSdss Product Owner
    ------------------------------



  • 8.  RE: Dfdss dump to minio S3 cloud?

    Posted Wed March 05, 2025 11:28 PM

    Hi,

    Thanks, I've managed to dump several data sets to our internal cloud servers:

    I've noticed CDA is adding compression support in https://www.ibm.com/support/pages/apar/OA66536 ,

    is there any future plans to allow compression with  DFDSS dump to cloud?

    Thanks,

    Travis



    ------------------------------
    Travis Thorne
    ------------------------------



  • 9.  RE: Dfdss dump to minio S3 cloud?

    Posted Thu March 06, 2025 11:30 AM

    Hi Travis, 

    Im glad you were able to use our new support and hope that the setup and invocation was fairly simple - especially since you had already used GDKUTILS.

    As far as compression, you will be glad to hear that you will not have to wait very long.  Please review the 1Q25 announcements, here: https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/fiona-king1/2025/02/18/zos-31-1q-2025-enhancements?communityKey=2eba32a4-9426-4331-81e7-8ea120e4fd99. Specifically under "DFSMS cloud object storage enhancements,"  for OA67394.



    ------------------------------
    ERNESTO FIGUEROA DFSMSdss Product Owner
    ------------------------------



  • 10.  RE: Dfdss dump to minio S3 cloud?

    Posted Mon March 31, 2025 12:51 AM

    Hi Ernesto,

    Thanks, I just got the PTF for OA67394 and was able to use the new CDACOMPRESS keyword :)

    ``` 

    PAGE 0001     5695-DF175  DFSMSDSS 3.1 DATA SET SERVICES         2025.090 12:41                           
       DUMP FULL  INDYNAM($$SR1W)                -                                                            
            CLOUD(MINIOCR)                       -                                                            
            CONTAINER(SYSRES)                    -                                                            
            OBJECTPREFIX(SR1W)                   -                                                            
            ADMINISTRATOR                        -                                                            
            ALLDATA(*)                           -                                                            
            ALLEXCP                              -                                                            
            OPT(4)                               -                                                            
            CDACOMPRESS                          -                                                            
            CDAPROVIDERFILE                                                                                   
    ADR101I (R/I)-RI01 (01), TASKID 001 HAS BEEN ASSIGNED TO COMMAND 'DUMP '                                  
    ADR109I (R/I)-RI01 (01), 2025.090 12:41:18 INITIAL SCAN OF USER CONTROL STATEMENTS COMPLETED              
    ADR618I (R/I)-RI03 (01), CLOUD INFORMATION OBTAINED FROM CDA IS REQUESTING DIRECT                         
    ADR016I (001)-PRIME(01), RACF LOGGING OPTION IN EFFECT FOR THIS TASK                                      
    ADR006I (001)-STEND(01), 2025.090 12:41:18 EXECUTION BEGINS                                               
    ADR006I (001)-STEND(02), 2025.090 12:45:54 EXECUTION ENDS                                                 
    ADR013I (001)-CLTSK(01), 2025.090 12:45:54 TASK COMPLETED WITH RETURN CODE 0000                           
    ADR012I (SCH)-DSSU (01), 2025.090 12:45:54 DFSMSDSS PROCESSING COMPLETE. HIGHEST RETURN CODE IS 0000      

    ```

    Kind Regards,

    Travis Thorne.



    ------------------------------
    Travis Thorne
    ------------------------------



  • 11.  RE: Dfdss dump to minio S3 cloud?

    Posted Tue April 01, 2025 10:51 AM

    Im glad to hear it Travis! 



    ------------------------------
    ERNESTO FIGUEROA DFSMSdss Product Owner
    ------------------------------