Go on z/OS - Group home

What is Go (or Golang)

Go, often called Golang, is a statically-typed, compiled programming language designed at Google by Robert Griesemer, Rob Pike, and Ken Thompson. Go looks similar to C but it is memory-safe, it has garbage collection, structural typing, and CSP-style concurrency.

In the Stack Overflow Developer Survey 2022, it has ranked as the #4 most wanted programming language after Python, TypeScript, and JavaScript. It is also associated with the highest paying jobs.

Source: https://survey.stackoverflow.co/2022/#section-most-loved-dreaded-and-wanted-programming-scripting-and-markup-languages 

Rob Pike explains the main reasons for Go’s success in his talk Why Go Is Successful (aka Simplicity is Complicated). Go may not have every feature of other programming languages, but it has rather an excellent subset of powerful features that are simple to use. 

It is used by many companies to build new web services. It is also used to develop system utilities. Docker is one of the best-known projects that is developed in Golang. 

Go on z/OS

Golang works well on Linux on Z for a long time. In March 2021, an officially supported version of Golang has been released for z/OS, called “IBM Open Enterprise SDK for Go”. It started with version 1.16 and IBM keeps the port up to date. We can download 1.18 today.

It comes in two editions: SMP/E and Pax. The Pax edition can be downloaded and installed by anyone. If you want to try it, just visit the IBM Open Enterprise SDK for Go on z/OS page, press the Try Pax edition link, and follow the instructions.

Advantages of Go on z/OS

There are several advantages of using Golang on z/OS. If you have your application developed in Golang, it can benefit from the great reliability and resiliency of the zSystems platform when you run it on z/OS. 

Typical business applications have to access data that reside on the mainframe no matter what technology they are developed in. If such an application runs on the same mainframe or in the same sysplex that has the data, the application will benefit from colocation. The application typically accessed the database using ODBC driver using TCP/IP protocol over the network. The HiperSockets on z/OS provide a significant boost in the speed and reliability of such access. You can see improvement in orders of magnitude depending on the previous network topology.

This blog examines the benefits that Golang on z/OS brings to z/OS programmers as compared with another modern programming language that has been the only choice how to easily run many open-source tools on z/OS - Java.

Using existing libraries on z/OS is easy in Java but it has been a problem for C or C++ since building such projects on z/OS is not straightforward because of differences between various C/C++ compilers and missing or different build toolchains on z/OS. With Golang on z/OS, you can use a vast collection of open-source libraries easily, and get native binaries on z/OS.

Where it makes sense to use Go over Java on z/OS

Java code is compiled into an intermediary form that is called bytecode. Such code is packaged in JARs and other types of archives and is not dependent on any platform. Virtual machine in Java Runtime Environment on a particular platform will compile such bytecode in native code before running the Java application (JIT - just in time compilation) or it can interpret it. Any JAR can be used on z/OS so the effort to use many 3rd party libraries is very low. But there is an overhead of the JIT compiler and of the virtual machine. In the case of a long-running application or server, the overhead is not critical but if your application is a small tool that runs for a short time but executes frequently, the performance will not be ideal in Java.

In this case, it makes sense to leverage a language that compiles into native code that has a lightweight runtime environment. Golang is such a language. On top of that, you can take advantage of many 3rd party libraries, and the final result is built into a single file that can be easily distributed.

You do not need any additional runtime environment to be installed on z/OS as in the case of Java, Node.js, and Python. You just copy the single file and you are ready to use go!

Example of a command-line tool

We will show some of the benefits listed above on an example of a useful command-line utility developed in Golang. We have developed the same utility in Java as well to compare them.

md5sum is a utility that calculates and verifies 128-bit MD5 hashes, as described in RFC 1321. Such a hash is used as a small digital fingerprint of a file. As with all such hashing algorithms, there is theoretically an unlimited number of files that will have any given MD5 hash. However, it is very unlikely that any two non-identical files in the real world will have the same MD5 hash unless they have been specifically created to have the same hash. It is typically used to verify the integrity of the file and the md5sum utility is installed on almost every system with Linux. But it is not on z/OS. 

This utility is used as a part of the file synchronization process between z/OS and a workstation. MD5 hashes are calculated on both ends. This helps to identify different files quickly. No data need to be transferred for comparison. 

We will build it completely on z/OS but you can use some IDE such as VS Code with Zowe Explorer too.

Now navigate to where you build your projects and create a new directory:

mkdir md5sum-go
cd md5sum-go

Before we start adding code let’s initialize our project (please replace it with your URL - for example on your company’s GitHub Enterprise):

go mod init github.com/plavjanik/md5sum-go

This simple program will have just one source code file: main.go. We need to set the code page tag correctly so it is written and read in ASCII:

touch main.go
chtag -t -c ISO8859-1 main.go
vi main.go

We will paste a short source code:

package main

import (
	"crypto/md5"
	"fmt"
	"io"
	"io/ioutil"
	"os"
	"path/filepath"
	"regexp"
	"strings"
)

func main() {
	if len(os.Args) < 2 {
		help()
	} else if os.Args[1] == "-h" || os.Args[1] == "--help" {
		help()
	} else if os.Args[1] == "-c" || os.Args[1] == "--check" {
		if len(os.Args) == 2 {
			help()
		}
		checkHashesInFile(os.Args[2])
	} else {
		determineHashForFile(os.Args[1:])
	}
}

func checkHashesInFile(argFile string) {
	md5bytes, err := ioutil.ReadFile(argFile)
	if err != nil {
		error(fmt.Sprintf("failed to read file %s: %v", argFile, err))
	}

	splitBySpace := regexp.MustCompile(`\s+`)
	failedFiles := 0
	totalFiles := 0

	for _, line := range strings.Split(strings.TrimSuffix(string(md5bytes), "\n"), "\n") {
		columns := splitBySpace.Split(string(line), -1)
		if len(columns) != 2 {
			error(fmt.Sprintf("could not parse check file: each line should have two columns 'md5_hash file_name' but it is: '%s'", line))
		}

		argPath := filepath.Dir(argFile)
		fullPath := ""
		if len(argPath) > 0 {
			fullPath = argPath + string(filepath.Separator)
		}
		fullPath += columns[1]
		md5sum := md5hash(fullPath)

		if md5sum == columns[0] {
			fmt.Printf("%s: OK\n", columns[1])
		} else {
			fmt.Printf("%s: FAILED\n", columns[1])
			failedFiles += 1
		}
		totalFiles += 1
	}

	if failedFiles > 0 {
		fmt.Printf("md5sum: WARNING: %d of %d computed checksums did NOT match\n", failedFiles, totalFiles)
		os.Exit(1)
	}
}

func determineHashForFile(files []string) {
	for _, arg := range files {
		md5sum := md5hash(arg)
		fmt.Printf("%s %s\n", md5sum, filepath.Base(arg))
	}
}

func md5hash(file string) string {
	f, err := os.Open(file)
	if err != nil {
		error(fmt.Sprintf("failed to open file for read: %v", err))
	}
	defer f.Close()

	h := md5.New()
	if _, err := io.Copy(h, f); err != nil {
		error(fmt.Sprintf("failed to process file content with MD5 hashing function: %v", err))
	}

	return fmt.Sprintf("%x", h.Sum(nil))
}

func help() {
	fmt.Println("Usage: md5sum [<option>] <file> [<file> [...] ]")
	fmt.Println("       md5sum [<option>] --check <file>")
	fmt.Println()
	fmt.Println(" -c, --check <file>   Check MD5 sums from <file>")
	fmt.Println(" -h, --help           Display this help message and exit")
	fmt.Println()
}

func error(errMsg string) {
	fmt.Println("Error:", errMsg)
	os.Exit(1)
}

The source is available at: https://gist.github.com/plavjanik/ffb611e87affe7c8cc7744025d14cb9

Golang has all the tools included so you compile and link the program using:

go build

You can see that there is a new file: md5sum-go

You can get an MD5 hash for your files using:

./md5sum-go main.go go.mod > files.md5

The files.md5 contains:

ca9492c440d207f4c90c145a98d77896 go.mod
b5c2a34989010b7fec225d3b36d9ec87 main.go

You can check that hash is still correct by:

./md5sum-go --check files.md5

go.mod: OK
main.go: OK

If you look closely at the code, you can see that it is importing only standard Golang libraries that enable you to calculate MD5 hash for data that you read from files. Regular expressions and various text processing functions are available for you.

Performance

So we can have a simple program in Golang that works well on z/OS and you can see that it works quite fast. We have done a more detailed comparison with the same program in Java.

The tests measured 1000 executions of CLI utility that computed MD5 sum for:

1. One 300 kB file
2. Ten 300 kB files (3 MB)
3. One hundred 300 kB files (300 MB)

We measured elapsed time (real), CPU time (time): usr + sys, and other metrics (zIIP time, I/O operations).

Results

Golang has beaten Java implementation in all cases (the elapsed time was 7 to 3 times lower and GP 4.5 to 2.5 usage times lower, and I/O count 13.5 to 9 times lower)*. With a small file (300 kB) the execution time for Golang was about 130 ms while almost 900 ms for Java. 
The benefit of Java running on zIIP can be seen if the program is processing a large amount of data (gigabytes) in Java (not in C via JNI).

Conclusion

Golang is ideal, among other things, for writing CLI utilities that are either running for a short time (less than 1 second) and/or calling native code. For such CLI utilities, the Golang program provides faster execution and lower cost for customers than the Java one. In future blogs, we will use Golang to call z/OS APIs to interface with JES.


*Disclaimer: Performance results shown was obtained in a controlled, isolated environment using an internal test suite. Performance of other workloads may vary.