A Game Changing Series part 6 - Cloud Data Access (CDA)
CDA Cornucopia of Enhancements
If you haven’t seen the rest of the Game Changing Series Blog posts you can check them out here to get a high-level explanation of CDA and the other enhancements!
The CDA team has been hard at work, ignoring their Holiday preparations and instead, preparing a cornucopia of plentiful enhancements for you!
1 GDKUTIL Enhancements
1.1 Long Object Name
One of the requests we’ve gotten is to allow continuation characters or some way to specify an object name longer than 72-80 characters long. Of course, you could create a data set with a big LRECL, and put the long name in the data set, but just because you can do a thing, doesn’t mean that it is easy to use.
GDKUTIL now recognizes a new DD name called, OBJNAMEX. When you include the OBJNAMEX DD, GDKUTIL will read each line of the DD to build the full object name. The leading and trailing whitespace from each line is discarded, and the remaining characters are put together to create the long object name. Of course, any blank characters in the middle of the text are retained, and you can’t exceed the maximum 1024 characters for an object name.
Our hope is that the OBJNAMEX support makes your work specifying object names easier. Below are two examples showing usage of the OBJNAMEX DD.
The example below shows imbedded blank characters within the object name. (The indentation is intentionally not lined up to demonstrate there is no need for beginning/ending in a specific column or character position. Additionally, you do not need to start every line with a forward slash (/) pseudo-directory indicator, I just thought it might make more sense to users.)
|
Example JCL using OBJNAMEX
|
|
//STEP001U EXEC PGM=GDKUTIL,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
UPLOAD PROVIDER(IBMCOS)
/*
//OBJNAMEX DD *
/bankbucket
/SYSTEM-BA
/Pricing_Insight_applicationID
/testlogging_from migration testing
/j5a99bg-ahe995ot110-dd8fy
/todays_time_stamp
/pricing-insight-log.log
/*
//LOCNAME DD *
OA65990.PRICING.INSIGHT.LOG(-1)
/*
|
|
5695-DF124 DFSMSdfp CDA 3.1 CLOUD OBJECT UTILITY 2024323 15:01:11
GDKU0013I OBJNAME: /bankbucket/SYSTEM-BA/Pricing_Insight_applicationID/testlogg
ing_from migration testing/j5a99bg-ahe995ot110-dd8fy/todays_time_stamp/pricing-
insight-log.log
UPLOAD PROVIDER(IBMCOS)
GDKU0013I LOCNAME: OA65990.PRICING.INSIGHT.LOG(-1)
GDKU0100I UPLOAD PROCESSING SUCCESSFUL
|
The example below shows using System Symbols within the instream DD to customize the object name.
|
Example JCL using System Symbols
|
|
//STEP001U EXEC PGM=GDKUTIL,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
UPLOAD PROVIDER(IBMCOS)
/*
//OBJNAMEX DD *,SYMBOLS=EXECSYS
/&SYSBUCK
/applicationID
/&SYSNAME
/j5a99bg-ahe995ot110-date
&YYMMDD-&LHHMMSS
/10mBds.txt
/*
//LOCNAME DD *
OA65990.SMSAU001.LREC1024.SAM.S10MB
/*
|
1.2 Text Conversion Overrides
CDA originally only performed text conversions from IBM-1047 EBCDIC to ISO8859-1 ASCII; We have received enthusiastic requests to provide more options for text conversions so this enhancement will let you choose the code pages used when converting text. The GDKUTIL batch JCL utility has been enhanced to allow you to specify the codeset name for the local text and the codeset name for the remote text.
The existing CONVERT keyword for GDKUTIL now additionally allows sub-parameters to be specified in the format: CONVERT(<local_codeset>,<remote_codeset). If you continue to specify the CONVERT keyword by itself, you will get the default values of IBM-1047, and ISO8859-1, unless overridden in the provider file for that cloud provider. You can specify CONVERT(<local_codeset>), and you will get the default remote codeset.
This support uses a concept of a local codeset and a remote codeset, so that CONVERT on UPLOAD will convert from local to remote. A CONVERT on DOWNLOAD will convert from remote to local.
The allowed codeset names are those accepted by the iconv() function. A sample list can be found in Table 1 in the z/OS XLC/C++ Programming Guide in the Internationalization: Locals and Character Sets chapter: https://www.ibm.com/docs/en/zos/3.1.0?topic=cscu-universal-coded-character-set-converters
|
Example JCL Showing Download of Text data that is encoded in UTF-8 to IBM-278
|
|
//STEP001U EXEC PGM=GDKUTIL,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
DOWNLOAD PROVIDER(IBMCOS)
CONVERT(IBM-278,UTF-8)
/*
//OBJNAMEX DD *
/demo-convert-download
/applicationID/j5a99bg-ahe995ot110-date/local-log.txt
/*
//LOCAL DD DISP=(NEW,CATLG),DSN=OA65990.CONVERT.DEMO.LOCALLOG,
// SPACE=(CYL,(5,5),RLSE),LRECL=1024,RECFM=VB,
// BLKSIZE=27756
|
1.3 Credential Management
We have heard our users loud and clear: credential management is an ever-increasing concern. Many security departments require cloud credentials to be changed every 30 days. As you multiply that out by each RACF user ID times the Sysplexes that need to have the credentials saved, it quickly becomes overwhelming. A near-term easy fix is to give GDKUTIL users the ability to perform credential management via batch JCL. We are expecting that DevOps pipelines can be used to submit batch JCL that will add new credentials via this method.
The GDKUTIL utility has been enhanced to add a new CREDENTIALS command with the three sub-parameters, ADD, DELETE, and LIST. Additionally, a new keyword, ZUSERID(<userid>), is recognized for the CREDENTIALS command, allowing the credentials of that z/OS UserID to be managed. (Of course, the z/OS userid associated with the JOB must have adequate permission to the <userid> key file as well as the keylabel in the CSFKEYS class.
The CREDENTIALS command will perform the credentials operation for the named PROVIDER (and optional BUCKET/OBJNAME).
1.3.1 CREDENTIALS(ADD)
Saving new credentials can be tricky because we don’t want to simply pass sensitive data like cloud credentials in JCL. One of the most common methods that DevOps pipelines use to transfer or save credentials in a remote system is by copying a file to the remote system, and then telling the credential management to use the contents of the file. (It’s bad practice to pass the credentials on the command line since that is often logged.)
When the CREDENTIALS(ADD) command is specified on the GDKUTIL SYSIN, GDKUTIL will look for a new DD named, CREDSNAM. This DD will hold the name of a z/OS UNIX file or data set that contains the credentials to be added for the named PROVIDER.
The contents of that z/OS UNIX file or data set will be a JSON document containing key:value pairs for the credentials you want to save. Currently, GDKUTIL, accepts the following keys:
· accessKey – The value is the S3 HMAC Access Key ID used in the authentication.
· secretAccessKey – The value is the S3 HMAC Secret Access Key used for authentication.
· tenant – The tenant value used in the authentication.
· userid – The userid value used for authentication.
· password – The password value used for authentication.
· accessURL – The Access URL used during authentication.
· credentials-file – The absolute path to the UNIX file that contains the JSON credentials obtained from the cloud provider Note: this is only used with OAUTH_2 authentication models such as Google Cloud Platform (GCP). This value may override a specified accessURL. Both may not be specified in a JSON credentials document.
|
Example Credentials JSON saved in z/OS UNIX file or data set
|
|
{
"accessKey": "myIBM_Cloud_AccessKey",
"secretAccessKey": "myIBM_Cloud_SecretAccessKey"
}
|
|
{
"credentials-file": "/u/ibmuser/GCPcreds/credentials.json"
}
|
|
{
"accessURL": "https://myauthserver.cloud.ibm.com/v1/auth",
"tenant": "New tenant name",
"userid": "cloud-server_userid",
"password": "CompletelyFakePassw0rd"
}
|
Below is example JCL showing saving credentials for the CLDTASK userid by the Service ID: STARTASK. The credentials to be added are contained in the add_S3_creds.json file in the format like the examples above.
|
Example JCL Adding default credentials for the IBMCOS provider for user CLDTASK
|
|
//CREDS001 JOB (XXXX),'MOVE CLOUD',MSGLEVEL=(1,1),
// MSGCLASS=H,USERID=STARTASK
//EXEC PGM=GDKUTIL,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
CREDENTIALS(ADD) PROVIDER(CLOUD1)
ZUSERID(CLDTASK)
/*
//CREDSNAM DD *
/devops/creds-agent/add_S3_creds.json
/*
|
|
Example JCL Adding default credentials for yourself
|
|
//CREDS001 JOB (XXXX),'MOVE CLOUD',MSGLEVEL=(1,1),
// MSGCLASS=H
//EXEC PGM=GDKUTIL,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
CREDENTIALS(ADD) PROVIDER(CLOUD1)
/*
//CREDSNAM DD *
/u/myuser/new_creds.json
/*
|
|
Get into the Weeds Note: Details about Credentials Resource
|
|
DFSMSdfp CDA has a concept of credentials applying to a particular S3 bucket or “resource”. When CDA is preparing to communicate with a cloud server, it first searches for credentials that match the bucket for the request. If none are found, then it looks for the default credentials, which are identified by the forward-slash character.
You may have noticed on the z/OS Cloud Data Access Authorization Utility ISPF application that the first panel has an editable field named, “Resource”. It is pre-populated with a forward-slash character, which is the default credentials identifier. You could have been using this field to identify a particular bucket that has different credentials than the default.
A real-world example of this is where you have a service account allowing access to the object storage used by your company. However, you want to share some data with the object storage account for someone else. In this case, because bucket names are unique across the entire world, the other person could create a bucket specifically for sharing data, and additionally create credentials for you to use to access that bucket. You would save those credentials by specifying the provider name as usual, along with the Resource set to the bucket name for the other account. This would allow you to then access the other account’s bucket. (Another, more straightforward way to accomplish this would be to just have another provider file with a different name and set those credentials to the shared ones.)
However, I have come across some applications that are using CDA’s Credential Management APIs to securely store sensitive information for many users. To do this, they use an identifier for each user as the “Resource”, along with the same provider name. That allows them to retrieve the credentials for each specific user easily, as they are stored in the same key file, encrypted by AES256 keys all under the application’s RACF userid.
|
1.3.2 CREDENTIALS(DELETE)
When you add credentials to the CDA key file, the old credentials are first removed, including the ICSF key label and encryption key that was used to secure those credentials; However, we figured it would still be good to give you the option to delete credential entries that you don’t want.
When the CREDENTIALS(DELETE) command is specified on the GDKUTIL SYSIN, attempt to delete the credentials entry for the named PROVIDER. The specific credential entry is identified by the combination of the provider name and optional resource name found on the OBJNAME/OBJNAMEX/BUCKET DD statement. If none of those DD statements are found, then the default credential entry of / (forward slash) is deleted. When deleting the credential entry, the associated ICSF keylabel and encryption key is also removed from ICSF.
Additionally, the ZUSERID keyword may be used to specify the RACF userid that the credential entry should be deleted from. The RACF userid associated with the JCL JOB must have ALTER access to the ZUSERID’s keylabel in the CSFKEYS class as well as write permission to the ZUSERID’s gdkkeyf.json key file.
|
Example JCL Deleting default credentials for the IBMCOS provider for user CLDTASK
|
|
//CREDS002 JOB (XXXX),'MOVE CLOUD',MSGLEVEL=(1,1),
// MSGCLASS=H,USERID=STARTASK
//EXEC PGM=GDKUTIL,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
CREDENTIALS(DELETE) PROVIDER(CLOUD1)
ZUSERID(CLDTASK)
/*
|
1.3.3 CREDENTIALS(LIST)
We felt it was important to round out the GDKUTIL credential management support by giving you the ability to list what credential entries currently exist in the gdkkeyf.json key file. This function only lists the credential entries identified by their resource name. The ZUSERID keyword may be used to specify that the list should be of credentials for a different user. The RACF userid associated with the JCL JOB must have read permission to the ZUSERID’s gdkkeyf.json key file. (Since nothing is being decrypted, this processing doesn’t need any authority to the CSFKEYS class. Additionally, we thought it would be a useful feature to allow you to request a list of all credential entries for all providers found in the gdkkeyf.json file.
|
Example JCL listing the credential entries for IBMCOS
|
|
//STEP001L EXEC PGM=GDKUTIL,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
CREDENTIALS(LIST) PROVIDER(IBMCOS)
/*
|
|
5695-DF124 DFSMSdfp CDA 3.1 CLOUD OBJECT UTILITY 2024287 15:55:06
CREDENTIALS(LIST) PROVIDER(IBMCOS)
LOG(DEBUG)
GDKU0116I LIST OF CREDENTIALS FOR PROVIDER IBMCOS
BUCKET SAVE DATE
/plxtest/ 2023-03-30 16:19:18
/ (All Buckets) 2024-10-12 14:31:47
GDKU0100I CREDENTIALS(LIST) PROCESSING SUCCESSFUL
|
|
Example showing a listing of credential entries for all Providers
|
|
//STEP002L EXEC PGM=GDKUTIL,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
CREDENTIALS(LIST) PROVIDER(*)
/*
|
|
5695-DF124 DFSMSdfp CDA 3.1 CLOUD OBJECT UTILITY 2024287 15:55:06
CREDENTIALS(LIST) PROVIDER(*)
GDKU0116I LIST OF CREDENTIALS FOR PROVIDER *
BUCKET PROVIDER NAME
/plxtest/ IBMCOS
/ (All Buckets) IBMCOS
/ (All Buckets) EMPTY
/ (All Buckets) GCP
/testbucketallcreds GCP
/ (All Buckets) GDKGCPS3
/ (All Buckets) DFLINUX1
/ (All Buckets) IBM
/ (All Buckets) COSCHAR
/temprc/ GOOGS3
/ (All Buckets) GOOGS3
/asdf8shdfsahdfkjsahskjdhf2kjsahfdkhsdf AZURE
/ (All Buckets) AZURE
/ (All Buckets) Dropbox
/ (All Buckets) PLPSC
/ (All Buckets) S3NIMBUS
/ (All Buckets) AWS
/ (All Buckets) PLPSCDS
/ (All Buckets) DFLINUXRO
/ (All Buckets) IBMCOSLIST
sdpfu;lkasjdf2l4kj CLOUDS3REST
newaccesskeyfirhtis CLOUDS3REST
/ (All Buckets) IBMCOS2
/ (All Buckets) DFLINUX2
/ (All Buckets) DFSMSIBMCOS
/ (All Buckets) IBMCOSCDA2
/ (All Buckets) IBMCOSOAM
/ (All Buckets) AZUREFILE
/share-bucket/ SHARE.DEMO
/ (All Buckets) IBMCOSLISTPART
/bucket TESTCLOUD
/ (All Buckets) TESTCLOUD
/ (All Buckets) FAKECLOUD
GDKU0100I CREDENTIALS(LIST) PROCESSING SUCCESSFUL
|
1.4 Next!
Don’t fret API users, we didn’t forget you with this cornucopia of enhancements! Check back on December 3rd 2024 for A Game Changing Series Part 7 covering the API Enhancements!
All of these enhancements are being made available on z/OS v2r5 by the PTFs for OA65990. Publication updates (there’s a lot) can be found at: https://public.dhe.ibm.com/eserver/zseries/zos/DFSMS/CDA/OA65990/OA65990_Publication_Updates.pdf
Check out our Cloud Data Access content solution page to learn more and get started!
We would love to hear your thoughts on CDA and these new enhancements! Leave a comment on this blog post, start a discussion on the DFSMS Community Page, or join Mainframe Data Management LinkedIn group to discuss there!
Author:
Andrew Wilt
Editor:
Alexis Kapica (Ricci)