Db2 for z/OS and its ecosystem

 View Only

Db2 Analytics Accelerator on Z IFL sharing and LPAR weights for multi-node Accelerators explained

By MEHMET CUNEYT GOKSU posted Mon November 01, 2021 08:50 AM

  

A technical article by Roland Seiffert
Roland is a recognized expert on all aspects of Db2 Analytics Accelerator on IBM Z. He has been the inventor, architect and technical leader for the Accelerator on Z at IBM Research and Development in Böblingen, Germany. Besides his development leadership, he has worked intensively with clients world-wide to design and implement solutions based on this technology. He has also been a frequent-speaker and instructor at SHARE, IDUG, and IBM TechU.

Larger deployments of the Db2 Analytics Accelerator on Z use “multi-node setups”, i.e., multiple LPARs that work together as a single Accelerator. This is mostly because the database engine used is designed for scale-out. There is a single topology for all sizes: one HEAD node and five DATA nodes. Each node is represented by a single LPAR. The HEAD node receives requests from Db2 z/OS, such as queries or load jobs – and distributes the actual work to the DATA nodes which hold and process all user data.

This makes the required LPAR configuration for a system running the Accelerator more complex. In this article, I’ll explain how to set up processors and weights for a multi-node Accelerator without affecting the pre-existing LPARs.

The technical background for this article is described in detail in the book “Processor Resource/Systems Manager Planning Guide” (IBM order number SB10-7175-01c) which can be downloaded at

https://www.ibm.com/support/pages/processor-resourcesystems-manager-planning-guide


Basics

While smaller single-node configurations often use dedicated IFLs, multi-node setups use shared IFL processors. Why? There are two major reasons:

1.  The HEAD node needs little resources most of the time. For example, during query processing, it will only receive the query and distribute it to the DATA nodes, and later, collect and deliver the results – all the heavy lifting is done by the DATA nodes. Dedicated resources for the HEAD node would be “idle” most of the time and not be available for actual query processing.

2. The workload on the DATA nodes is not perfectly balanced. The platform hypervisor PR/SM is able to use processor power optimally where it is needed most.

 

Let’s consider an example. Assume that 36 IFL processors are available for multi-node deployment. If dedicated IFLs would be used, each node would have 6 IFLs. Query processing per DATA node could leverage 6 processors. If CPU utilization is 80% on average, net capacity used is 4.8 IFLs per DATA node.

 

With shared IFLs, the assignment following standard guidance would be 9 (shared) IFLs per LPAR/node, all LPARs are put into an LPAR group with absolute capping set to 36. Simplifying a bit – reality is not completely perfect – during query execution, HEAD would need no resources. This would now leave all cycles of 36 processors to the DATA nodes, i.e., 7.2 IFLs per DATA node – 20% more than with dedicated IFLs. The configuration of the DATA nodes uses 9 logical processors, however. As DATA nodes vary in CPU utilization during processing with, e.g., 80% average as in the dedicated example above, theoretically 7.2 processors (80% of 9 logical processors) could be actually used.

 

In summary, using shared IFLs may improve (query) processing up to 50% by increasing utilization from 4.8 to 7.2 processors per node. Note that in reality, benefits may be smaller – but they are always substantial. Also, if there is a mixed workload with load jobs and/or incremental update running in parallel to query processing, the benefits are reduced because the HEAD node will require some of its assigned resources itself.

LPAR group and capping

All LPARs constituting a multi-node Accelerator are put into an LPAR group with absolute capping which represents the Accelerator. Only the HEAD node is “visible” externally and holds the connection to Db2 z/OS. Absolute capping limits the processor usage to the sized and licensed capacity – in the example 36 processors – for the group of LPARs. This is important to avoid impacting other (shared) IFL workloads on the same system and to enforce the limits as licensed, since the Accelerator license is per PVU (“processor value unit”). Without capping, the sample configuration could use up to 6x 9 IFLs, i.e., 54 IFLs if they are available and not currently used by other workloads.

 

Remark: if there is more (shared) IFL workload on the system, it is recommended to apply absolute capping to the other LPARs as well to avoid that the capacity added for the Accelerator can be used by the pre-existing workload.

 

LPAR weights

While capping limits resource utilization, it does not provide any guarantee that CPU resources are available for the Accelerator. This can be achieved by defining appropriate LPAR weights.

 

Background: A CPC may have many physical IFL processors defined and activated. Some LPARs will use dedicated IFLs which are directly taken from the physical IFLs when activating the LPAR. The remaining physical IFLs – not used by LPARs with dedicated processors – constitute the shared IFL pool. There is only a single such pool on the whole CPC. The hypervisor PR/SM will use the physical processors from the pool to provide CPU resources to the logical processors defined in LPARs using shared IFLs.

 

For each LPAR using shared IFL processors, there is a definition of the

  1. number of logical processors
  2. LPAR weight

 

While the sum of logical IFLs may be higher than the number of physical IFLs in the pool, the maximum number of logical IFLs per LPAR is the same as the size of the pool.

 

The LPAR weight determine the LPAR share of IFL resources relative to all LPARs that use IFLs from the shared IFL pool. This determines the processor guarantee for the LPAR, i.e., the amount of physical IFL resources scheduled to the logical processors of the LPAR.

Example: Shared IFL pool has 30 physical processors, there are 4 LPARs defined

  1. 16 logical processors, weight 158 -->  share 0.53, guarantee 15.80
  2. 10 logical processors, weight 62   -->  share 0.21, guarantee 6.20
  3. 8 logical processors, weight 20     -->  share 0.07, guarantee 1.98
  4. 6 logical processors, weight 60     -->  share 0.20, guarantee 5.95

The sum of LPAR weights is 300. The resulting LPAR shares and processor guarantees are in blue above.

In general, the processor guarantee of an LPAR should be less or equal to the number of logical processors assigned to the LPAR. Also, if absolute capping is used, the processor guarantee should be less or equal to the capping limit.

 

On the other hand, an LPAR can obtain more than its processor guarantee if other LPARs don’t need (all) their resources. The maximum capacity is the lower of the number of logical processors and an absolute capping (if set). The processor guarantee will always be made available if needed, even if all other LPARs run at full utilization. In that case, cycles will be stolen to fulfill the other LPARs’ guarantee.

 

Another aspect of LPAR weights is that they strongly influence the “placement” of processor resources, i.e., which physical IFL will be used by which logical processor. PR/SM needs to solve the problem that it chooses an allocation that 1) allows it to deliver on the processor guarantee, and 2) is as “optimal” as possible, e.g., keep multiple processors on the same chip close to memory. PR/SM takes the LPAR weight as an indication of importance and solves the problem iteratively in the order of decreasing LPAR weight. In other words, the higher the weight is, the better the placement will be – at least initially (PR/SM may move processors and even memory during execution). The whole process is very complex…

 

Putting it together: When defining LPAR weights for a new Accelerator, we want to achieve the following goals

 

  1. The sum of processor guarantees of HEAD and DATA LPARs should be the sized capacity, i.e., the number of IFLs determined by the sizing exercise for the Accelerator.
  2. The capacity of the Accelerator should be capped at the sized value.
  3. The pre-existing LPARs receive the same capacity as before without any change in their LPAR configurations, including logical processors and weights.

 

Such a setup ensures that and the new Accelerator gets the desired (sized) capacity while at the same time, existing applications are protected from being impacted by the new Accelerator and vice versa.

The first step is therefore to determine the required sum of LPAR weights for the whole accelerator, i.e., HEAD plus 5 DATA LPARs. Assume that the pre-existing setup has a shared IFL pool of P0 physical IFLs and the sum of all LPAR weights using the shared pool is W0. We add an Accelerator with a sized capacity of C IFLs and therefore add C physical IFLs to the system and the shared pool. We want a processor guarantee of C for the Accelerator, so we can calculate the LPAR share.

Now we can determine the required total weight WT for the Accelerator’s LPAR group:

The next question is how to assign the weights to the individual HEAD (WH) and DATA (WD) LPARs. For the Accelerator, we want the best placement for the DATA nodes, since they do the very compute-intense heavy lifting. Therefore, the LPAR weight of the DATA nodes has to be higher than that of HEAD. The current IBM recommendation[1] is to give HEAD 10% of the overall weight and to distribute the remaining 90% equally among the 5 DATA nodes, i.e., each DATA LPAR will get 18%. This ensures that DATA nodes are placed first -optimally- and HEAD is “squeezed in”.

In general, this should provide the best solution. There may be exceptions with very specific workloads, e.g., massive, time critical load jobs running concurrently with queries. In such a case, it may be beneficial to increase the LPAR weight of HEAD above DATA. But these are rare circumstances and should be thoroughly investigated before making such changes.

Sample configuration

Now, let’s put everything together in a concrete example of a configuration. Let’s assume the following:

 

  • (existing) 18 physical IFLs
    • 8 dedicated IFLs used by 1 LPAR (A)
    • 10 shared IFLs used by 2 LPARs (B, C) with 8 logical processors each, with LPAR weights of 350 for B and 150 for C
  • (new) 36 physical IFLs for a multi-node Accelerator

 

In the existing setup, the shared IFL pool has 10 physical processors – the 8 dedicated IFLs are not “virtualized” and therefore not part of the pool. The sum of all LPAR weights for shared IFL LPARs is 500 which is the 100% mark and corresponds to the full 10 processors. LPAR B will be guaranteed 350/500=70% of the available IFLs in the pool, i.e., 7.0 processors. LPAR C will get 150/500=30%, i.e., 3.0 processors. Both LPARs are limited to a maximum of 8 processors because they have 8 logical processors defined.

Figure 1: Existing (pre-Accelerator) configuration

For the new Accelerator, 36 physical IFL processors will be added to the system which will be assigned to the Accelerator resulting in a total of 54 physical IFLs. The shared pool grows to 46 IFLs. We want to keep the LPAR definitions including the weights for the existing LPARs unchanged – and keep the processor guarantee for the two LPARs.

 

In order to achieve that, the sum of LPAR weights for the Accelerator needs to be calculated such that they reflect exactly 36 IFLs processor guarantee, or an LPAR share of 36/46 =  78%. The existing weights sum up to 500, the Accelerator needs a total of 1800 in weights.

 

Using the formula above

After calculating the total weight for the Accelerator, we need to determine the LPAR weights for the individual members of the group. The guidance here is to give HEAD 10% of the overall weight and to distribute the remaining 90% equally among the 5 DATA nodes, i.e., each DATA LPAR will get 18%. The result in the example is HEAD with an LPAR weight of 180 and each DATA with an LPAR weight of 324.

In summary, we can calculate the following configuration for the multi-node Accelerator:

Figure 2: Calculation of logical processors and LPAR weights for a multi-node Accelerator

 

The new complete configuration is as follows:

Figure 3: Final configuration after adding 36 physical IFLs and creating a multi-node Accelerator

This setup ensures that pre-existing LPARs receive the same capacity as before without any change in their LPAR configurations and the new Accelerator gets the desired (sized) capacity. At the same time, the setup protects the existing applications from being impacted by the new Accelerator and vice versa.

[1] This recommendation may change in the future since IBM is still optimizing the settings. They may also be workload-dependent. Therefore, It is better to  double-check with IBM. The good thing is that LPAR weights can be changed dynamically without any downtime. So they can be adjusted easily at a later point.



#Db2forz/OS
#IDAA
0 comments
35 views

Permalink