If you haven’t seen the rest of the Game Changing Series Blog posts you can check them out here to get a high-level explanation of CDA and the other enhancements!

Microsoft Azure support

The DFSMSdfp CDA team has been hard at work with a new enhancement in OA65925 that brings Azure Blob Storage support to z/OS v2r5! This functionality was already available in z/OS 3.1 but a customer requested it on z/OS v2r5. The customer also wanted to access the Microsoft Azure Files service from z/OS, so this enhancement also delivers a new provider file tailored to access that service from both z/OS v2r5 and z/OS 3.1.

This means you can use GDKUTIL (and direct API calls) to UPLOAD, DOWNLOAD, LIST and DELETE blobs in Azure blob storage on z/OS v2r5 systems. You can also use GDKUTIL (and direct API calls) to UPLOAD, DOWNLOAD, LIST, DELETE files from the Azure files service.

The Microsoft Azure Files service has some different concepts from object storage that you should be aware of: File Share, Directory, and File; Let’s go over some of those details below:

File Share

The highest level is called File shares. We think of it as similar to a UNIX file system. Within the file share, you will find directories, sub-directories, and files. The AZUREFILE.json sample provider file has a CREATEFILESHARE operation, so that you can use GDKUTIL to create a file share without needing to visit the Azure portal.

Directory

Within a file share you can create directories and sub-directories. The AZUREFILE.json sample provider file has a CREATEDIRECTORY operation, so you can use GDKUTIL to create directories as needed. A directory must first exist before you can create a file within it.

File

Lastly, a file in the File share must have been created before data being put in it. Not only that, We found out that the created file size must be known at the time it is created. (Andrew mistakenly thought we only needed to know the maximum size it may ever grow to. But when the file was created that way, it was actually a 4 TiB file with mostly zeroes. We didn’t notice until the bill came, which was the cost of a large LEGO set. Ouch!) We didn’t want you to run into the same thing, so the provider file has a CREATEFILE and CREATEANDUPLOAD operations which tell CDA to figure out the size of the file being uploaded. This unfortunately means that you need to take other actions if you want to add to the file later. 

With this continuous delivery support, more z/OS users can connect with their Azure storage and files used by the rest of the company. We are hoping this further integrates your z/OS environment with the rest of the business. There are more details to this support, so if you are interested in the nitty-gritty, see the CDA Azure Support Technical Deep-dive section below!

CDA Azure Support Technical Deep-dive

To get started, simply copy the sample provider file (AZURE.json) from /usr/lpp/dfsms/gdk/samples/providers/ to your CDA provider directory. Then by starting the z/OS Cloud Data Access Authorization Utility from ISPF Option 6, you can choose the AZURE provider, open the credentials entry panel, and save the storage account name and key to be used when accessing the Azure blob storage from z/OS.

Additionally, you can also access the Azure Files service from z/OS using GDKUTIL (and direct API calls). The setup steps are the same as above, with the exception that the provider file name is AZUREFILE.json.

After you have copied the AZUREFILE.json and AZURE.json sample provider files from the CDA samples directory (/usr/lpp/dfsms/gdk/samples/providers/) into your user home directory (~/gdk/providers/) you are ready to do more setup to talk to the Azure services.

You will need to save your Microsoft Azure Access keys in CDA. Below, I’ve included a screenshot showing the place where you can find the Access keys you will save in CDA. Remember the ‘Storage account name’ and click the “Show” button to see the Key. You can simply copy it into your clipboard to paste into the ISPF Panel later.

The screen below is the first panel displayed after using EX ‘SYS1.SAXREXEC(GDKAUTHP)’ from ISPF option 6. The panel is displaying that it found three provider files: AZUREFILE.json, AZURE.json, and IBMCOS.json. After typing the number 2 in the ‘Select Cloud Provider’ field, and hitting enter, the ‘Encryption Parameters’ display shows that AZURE is the provider that we will be saving the credentials for.

After typing the letter O on the Option ===> line and hitting enter, the Azure credentials entry panel is displayed. It is slightly different from the AWS4 credentials entry panel because it prompts for the Storage account name and the Access key. Enter your storage account name from the Microsoft Azure portal and paste the key into the Access key field. The TAB key may be used to navigate to the start of each field. Hitting Enter after putting a value into one field will also automatically position to the next field. As a security measure, any characters entered into the Access Key field are not displayed, and after the Enter key is pressed, asterisks are displayed for each character that was placed there.

Azure Blob Storage

Accessing objects in Microsoft Azure Blob Storage is very similar, conceptually, to accessing objects in S3 object storage. A Container is similar to an S3 Bucket; You need to create a container before you can place objects (Blobs) in it.

·       The GDKUTIL UPLOAD command will use the WRITEOBJECT operation to upload the data (Put Blob).

·       The GDKUTIL DOWNLOAD command will use the GETOBJECT operation to download a blob (Get Blob).

·       The GDKUTIL DELETE command will use the DELETEOBJECT operation to delete a blob (Delete Blob).

·       The GDKUTIL LIST command can list what containers are available – When OBJNAME is /, the LISTBUCKETS operation is used to get a list of Azure containers (List Containers).

·       The GDKUTIL LIST command can list blobs in a container. When OBJNAME is a container name /<container_name>/, the LISTOBJECTS operation is used to get a list of blobs (List Blobs).

·       The GDKUTIL LIST command can return information about a specific blob. When OBJNAME is a container name plus a blob name, /<container_name>/<blob_name>, the HEADOBJECT operation is used to get metadata about the blob (Get Blob Metadata).

Azure Files Service

File Share

You may be provided the name of a file share within your storage account that you should use. If not, you can create one using GDKUTIL with the OPERATION(CREATEFILESHARE) command. The OBJNAME you provide should be /<file_share_name>/ . Now that you have a file share, you can start placing files and directories in it.

Directory

As you upload files to the Azure File service, the forward-slash character in the name in OBJNAME for GDKUTIL UPLOAD operations is interpreted as directory level separators. For example, any forward slash characters after the file share name indicate a directory level.

e.g. /fileshare1/dir1/dir2/logs/file-output.log

All directory names in the OBJNAME must have been created before you UPLOAD the file. You can use GDKUTIL with OPERATION(CREATEDIRECTORY) and specify the directory name in the OBJNAME DD to create each directory.

Create File

Before sending data to a file in the Azure File service, that file must first exist. Not only that, the box for the file must have the size set to the exact size of the file. You can create the empty box for the file data using GDKUTIL with the OPERATION(CREATEFILE) command, specifying the LOCAL or LOCNAME for the z/OS UNIX file or data set you are going to upload later.

In order to set the size of the file, CDA must figure out how much data is going to be sent. If you are uploading a z/OS UNIX file, CDA can figure that out with a simple fstat() call. However, if you are uploading a data set, CDA will have to read the entire data set to figure out how many bytes are in it.

Upload (create and upload)

You can create and upload something from z/OS in one operation as well with GDKUTIL and the OPERATION(CREATEANDUPLOAD) command. This operation will first figure out the size of the file, and then upload its data to the File share.

List

Once you have put some files and directories in the File share, you probably want to list what is there. The GDKUTIL LIST command can do that. Specifying / as the OBJNAME will return a list of the file shares defined (using the LISTBUCKETS operation). Specifying /<file_share_name>/ will return a list of the directories and files in that top level. You can also specify directory names, ending in a forward slash, to see the list of directories and files in that directory level. Lastly, if you specify a file name (name doesn’t end in a forward slash), then attributes of that file will be returned.

Download

Downloading files from the Azure file service can be accomplished simply by using GDKUTIL and the DOWNLOAD command.

Delete

Deleting files from the Azure file service can be accomplished by using GDKUTIL with the DELETE command. The OBJNAME DD will hold the name of the File to be deleted.

You can delete multiple files with GDKUTIL by using the DELETE command with the BUCKET DD name specifying the directory name where files should be deleted. You can use the REGEX keyword to narrow down the files that are deleted.

Delete directory

Unfortunately, deleting a directory is not the same as deleting a file. (Just like UNIX rm deletes a file, but rmdir deletes a directory.) You would use GDKUTIL with the OPERATION(DELETEDIRECTORY) command to delete a specific directory. I think it will return an error if the directory is not empty, though.

Authentication Options: SharedKeyLite and SharedKey

During this project, we also made some changes relating to the Azure authentication that CDA supports. We learned a few things that didn’t stand out in the documentation, so we thought we’d share in case it helps someone else.

What Azure expects?

Azure expects that each request made against your resources on the Blob or File storage must be authorized for security purposes. This is what keeps your data secure, but it also means that we must do some extra work for each of the requests that we make.

For the authorization, Azure supports two methods: SharedKey and SharedKeyLite. CDA in z/OS 3.1 used to support only the SharedKeyLite, but now CDA will support both SharedKey and SharedKeyLite from v2r5 and above.

Let’s dig into what is required for these requests.

Authorizing your request with SharedKey:

·       Including the Date Header: All requests require that an UTC timestamp is present. So, we must send a x-ms-date header or Date header.

·       Constructing the Canonicalized Header string: All x-ms- headers need to be known for this portion of the process.. The work also involves converting these headers to lowercase, sorting them in lexicographic order, and some string manipulation such as removing any duplicates and leading/trailing blanks.

·       Constructing the Canonicalized resource string: To construct the resource string, it is required that the URI is heavily processed. This step involves retrieving query parameters from the URI, followed by converting them to lowercase, lexicographic sorting of query:value pairs, URL decode, handling duplicate keys and more. It also requires that we sort the query values lexicographically, if there are more than one value for a query.

·       Constructing the Signature String: For constructing the signature string, we first need to include the VERB, followed by the 11 specific headers, the canonicalizedHeaders and the canonicalizedResource.

·       The Auth Header: This calculated signature string is then encoded using the HMAC-SHA256 algorithm, followed by Base64 encoding of it.

Now that’s a lot of work but you don’t have to worry about any of it because CDA takes care of this for you!

The SharedKeyLite is a lighter version of the SharedKey. The major difference is in the number of headers used to form the signature string, which makes it slightly less secure than the SharedKey. Being “light”, it has a lower overhead and performs requests faster. So, we leave the choice to you!

Just specify the intended auth method (Default when not specified is SharedKeyLite), and CDA does all the work, so that your focus is on the request rather than its technical complexities.

CDA flow through a request to an Azure service.

If you made it this far, congratulations! We hope this new functionality helps with the things you want to do between z/OS and Microsoft Azure services.

These enhancements were delivered in APAR OA65925 and are available on z/OS 2.5 and above.

Check out our Cloud Data Access content solution page to learn more and get started!

We would love to hear your thoughts on CDA and these new enhancements! Leave a comment on this blog post, start a discussion on the DFSMS Community Page, or join Mainframe Data Management LinkedIn group to discuss there!

Stay tuned for ‘A Game Changing Series Part 5’!

Authors: 

Andrew Wilt

Mani Sankar Thirugnana Muthuvelan

Editor:
Alexis Ricci