In this blog article, I like to contrast Kubernetes to System Automation and show some of the similarities but also point out differences. I want to share with you what I have learned on my journey to containerization, and I hope that this will also help you to better understand how both orchestrators can work together once z/OS containers are available in future.

If you are new to this blog series or if you haven't heard about Kubernetes or System Automation before, I would recommend reading my first article to get started with containerization in general and where you can also find pointers to subsequent articles in the series.

Common goals

Both, Kubernetes and System Automation provide high availability, scalability and a declarative operations model for the workloads they are responsible for. However, their approaches are different.

The following table highlights similar terms and attributes of Kubernetes and System Automation.

Kubernetes deploys Pods within a cluster of nodes, specifically worker nodes. We leave the control-plane out of the comparison for this. Each Pod represents an instance of an application, for example a front-end webserver. It is the Kubernetes scheduler's task to determine which node is selected for deploying a Pod. In a first step, the scheduler determines the list of feasible nodes that meet specific scheduling requirements. An example could be that a node meets specific hardware features such as a GPU. In another example, a Pod may only be scheduled on a node where specific infrastructure services are running.

In a second step, the scheduler scores all feasible nodes to determine the node that is best suited for Pod placement. The scoring depends, for instance, on resource requirements or whether the container images are already available on a node. There is a lot more to say on the scheduler, but I keep that for a later article.

System Automation in contrast starts and stops resources on z/OS systems within a z/OS sysplex. A resource can be a single application, like a started task, for which System Automation uses the type APL. But it could also be a group comprising other resources. In this case, System Automation offers the type application group, or APG in short. All the resources controlled by System Automation and the relationships among each other are pre-defined in the automation policy.

One of the most important goals of System Automation is to ensure high availability. When a z/OS system is loaded initially (mainframers say IPLed), the resources need to be started as fast as possible in the right order to ensure that batch processing or online transaction processing can take place for the lines of business. Hence, System Automation uses "available" as the default desired status of any resource.

The counterpart of Kubernetes' scheduler is the System Automation Manager. It knows about each resource's status at any given time and determines what action needs to happen next by evaluating current status, constraints, conditions and relationships. An action is then sent as an order to the System Automation Agent who is responsible for the actual start or stop of that resource.

In many cases, System Automation doesn't have to make a placement decision. When the application is defined on a system, it will simply be started on it when it is desired to be available. In those cases, where System Automation must select a particular z/OS system, the automation policy contains rules that determine the preference of a resource. In contrast to Kubernetes that selects a node, System Automation selects a resource by its preference value. Only when preference values are equal, it makes a random decision.

On z/OS, the decision on which systems resources can run is normally made by the mainframe infrastructure team. For instance, they decide to start production workload preferably on the system with more CPU power and reserve the smaller one for backup purposes only. In this case, the preference value for a resource defined on the larger system is higher than the preference value for a resource defined on the smaller system. Hence, in the evaluation process, the System Automation Manager sends an order to start the resource on the larger system if this system is available.

If you look at this from out outside-in perspective, the net result is the same. The application is started on the system that has more CPU power unless that system is currently not available due to a planned or unplanned outage, in which case the other system would be selected. But it is important to understand this fundamental difference how the selection mechanism works internally.​

Pods, Deployments, Resources

Using the example above, a Pod could be a front-end webserver running as a container. This single-container application instance is the most typical use of a Pod. There are situations, however, when multiple containers share the same hostname, process, network and storage resources assigned to the Pod. When this is the case, the containers running inside the Pod are always co-located and their lifetime is the same as the lifetime of the Pod.

In System Automation, you would use an APL resource to represent a single started task and you would use a basic APG resource to represent two APLs that are co-located and managed together.

Our front-end webserver could be just one instance out of multiples in the sysplex. In Kubernetes, you would create a Service of type LoadBalancer that can balance the requests over multiple Pods. So, rather than managing these Pods individually, we would create a Deployment with an initial replicas value. When such a Deployment is created, Kubernetes starts as many Pods to meet the desired target as specified by the replicas value. When the number of requests increases, another webserver instance can be started by scaling out the Deployment: Increasing the replicas value changes the desired status of the Deployment and Kubernetes will respond by creating another Pod for the Deployment's ReplicaSet.

Also here, System Automation has a similar resource called server APG. This form of application group is intended to combine similar resources that all perform the same task and that can be spread within a single z/OS system or across systems within the z/OS sysplex. Like Deployments, a server APG can be scaled horizontally based on demand. If the so-called availability target is increased by one, System Automation selects another resource based on the resource's preference value. Likewise, resources are de-selected and stopped, if the availability target is decreased.

The picture below compares both solutions side by side:

The load balancing and routing itself is typically done outside of Kubernetes and System Automation. In case of Kubernetes, the cloud provider can provide a load balancer which distributes requests among a type LoadBalancer Service. On z/OS, it is the Sysplex Distributor which comes as part of the z/OS Communication Server and that can distribute incoming requests among those applications that have registered as a service following recommendations from the z/OS Workload Manager.

Declarative management and goal-oriented automation

One of the most pleasant characteristics of a tool like Kubernetes or System Automation is the fact, that you can set goals and you let the tool figure out the details.  But how do you do that?

Well, in Kubernetes, all the components have a specification, and the entire set of specifications is stored in a highly available key value store called etcd. Each specification defines the desired status of its component. If the specified component doesn't exist, it is created. Once created, it has an actual status. If the actual status matches the desired status within the specification, life is good and there is nothing to do for Kubernetes. But if the actual status differs from what the specification says, Kubernetes will adjust.

When the cloud administrator uses kubectl to change the replicas of a Deployment, for instance like so:

the new goal (replicas=10) is stored in the specification and when the actual number of replicas is smaller than 10, Kubernetes will create new Pods.  If a Pod fails for whatever reason, Kubernetes will detect that there is one less than desired and hence re-creates a new Pod.

Deployments are also the ideal tool to rollout new service. You simply specify the new image version that is used by the Pods in the Deployment. Kubernetes creates new Pods using the new image and replaces old Pods running the previous image. Example:

Once above command has completed, all the Pods in the nginx-deployment run a nginx image at version 1.16.1.

With System Automation, there are similar capabilities. All the resources and their desired status is maintained by the System Automation Manager and are persisted in a data set called the Takeover File. System Automation detects resource status changes and always compares the actual status against the desired status. Using a similar example as above, if the z/OS operator changes the availability target of a server APG like this:

the new availability target of the group SERV/APG is set to 10 and System Automation starts or stops resources in that group as needed to meet this availability target. If one of the resources in that group fails, System Automation attempts a restart in place if possible or selects another resource to reach the desired target again.

While System Automation cannot change the maintenance level of a resource in the group, it is able to recycle the resources one by one to pick up new maintenance that gets effective as soon as an application is restarted. The z/OS operator can do this with a single command:

INGGROUP SERV/APG ACTION=RECYLCE

Summary

Both, Kubernetes, and System Automation allow you to orchestrate a set of workloads using a declarative approach that enforces high availability and that supports elastic scaling as demanded by the business.

The most noticeable difference is that System Automation uses pre-defined resources and custom-specified preference values, whereas Kubernetes can create Pods on the fly on nodes that it selected and scored based on Pod requirements.

Now, System Automation likely won't ever compete with Kubernetes on scheduling containers on z/OS. But the question remains what that means to the existing operations when parts of the workloads are containerized, and others are not? This is what I like to start looking at in my next article.