This is the first blog entry of a series of blog entries about Integrated Synchronization, the new advanced data synchronization technique to process incremental updates to the accelerator. This blog entry provides an overview of Integrated Synchronization and describes the architecture and setup steps.
Integrated Synchronization overview
Integrated Synchronization is a new advanced data synchronization technique to process incremental updates to the accelerator. This functionality is integrated into Db2 for z/OS. Its purpose is to capture table changes from the Db2 for z/OS log and to apply these changes to the tables on the accelerator. For customers that want to use incremental updates it is no longer necessary to install and configure IBM CDC (Change Data Capture) of InfoSphere® Data Replication for z/OS®.
In addition, Integrated Synchronization provides the following advantages:
- Low latency
- Reduced CPU consumption on z/OS due to a streamlined and optimized design
- On z/OS, the workload to capture the table changes has been massively reduced and the remainder can be handled by IBM Z Integrated Information Processors (zIIPs)
- Simplified administration, packaging, upgrades, and support
- Enterprise-grade HTAP enabler: The integrated low latency protocol is now enabled to support significantly more analytical queries running against the latest committed data
- Supports replication of Db2 archive tables (not supported with CDC)
Db2 Analytics Accelerator V7.5 provides and supports both techniques:
- Incremental updates using Integrated Synchronization
- Incremental updates using CDC
Customers still have the choice which incremental update technique to use per Db2 subsystem that is paired to the accelerator. However, if the prerequisites can be met, the Integrated Synchronization technique is recommended for any new incremental update implementations. For existing incremental update implementations, migration from CDC to Integrated Synchronization is also recommended.
Prerequisites for Integrated Synchronization:
- Db2 12 for z/OS with APAR PH06628 (PTF UI63356) installed, running at function level V12R1M500
- Additional recommended APARs (available at the time of writing): PH19181, PH19886, PH20587, PH21187, PH21419
- Distributed data facility (DDF) with a secure port, configured for network encryption through AT-TLS
- For data sharing groups: Ensure it is possible to always connect to the same Db2 member, for example by using a specific secure port and location alias for Integrated Synchronization
- IBM Db2 Analytics Accelerator for z/OS Version 7.5.0 or later.
Integrated Synchronization architecture and setup
The following figure describes the components that are involved in Integrated Synchronization:
The Log data provider is a newly developed, internal Db2 for z/OS component that is provided with Db2 12 APAR PH06628 (PTF UI63356). It reads the Db2 log into a memory buffer via a service request block (SRB) that is scheduled in the Db2 address space DBM1.
The Log data processor is a newly developed, internal accelerator component. It is responsible for fetching the provided log data regularly to the accelerator into a staging area and applying the data from the staging area to the tables on the accelerator in an optimized high-performance way.
Compared to incremental updates with CDC, the design to read and fetch the log is streamlined, which results in reduced CPU usage and higher throughput.
For communication between both components the log data processor on the accelerator connects to Db2 for z/OS (DIST address space) via the DDF secure port. It authenticates to Db2 on z/OS with a z/OS user ID/password having MONITOR2 privilege in Db2 for z/OS on a special RACF DSNR profile (“ACCEL”); RACF PassTickets are also supported as an alternative. After the connection is established the log data provider starts reading the Db2 log and the log data processor fetches the provided log data into the staging area on the accelerator.
The log data transfer is always encrypted, which requires that AT-TLS be setup for this connection.
The setup of Integrated Synchronization is described in detail in the Db2 Analytics Accelerator Knowledge Center: https://www.ibm.com/support/knowledgecenter/en/SS4LQ8_7.5.0/com.ibm.datatools.aqt.doc/installmanual/topics/tp_idaa_inst_incr_updt_isync.html
The setup consists of the following steps:
- Installing the Db2 prerequisites
- Defining a secure network port for DDF
- Integrated Synchronization needs to maintain a stable connection to the same log data provider task on the same Db2 subsystem where the session was started.
- For data sharing groups: Ensure it is possible to always connect to the same Db2 member, for example by the following steps:
- Define a dedicated location alias and SECPORT for Integrated Synchronization on all Db2 members. In case you already use a SECPORT for other workloads the SECPORT for Integrated Synchronization would be a different one
- Start the location alias only on the Db2 member on which the Db2 log data should be provided for Integrated Synchronization
- This is shown in the following figure:
- AT-TLS configuration
- Create certificates, set up RACF keyring and store certificates
- Export public key of signer certificate in DER format and transfer it to the accelerator
- Setup AT/TLS to encrypt the DDF connections using the created certificates
- Prepare a user ID with required access rights for Integrated Synchronization
- Optionally use a RACF PassTicket or a password for authentication
- Required access rights: READ access to DSNR profile ssid DIST and ssid ACCEL, MONITOR2 privilege in Db2 for z/OS
- Enable Integrated Synchronization for a Db2 subsystem using the Db2 Analytics Accelerator Console
- Required input parameters: Db2 for z/OS IP address, DDF secure port, public key of signer certificate, prepared user ID
After the described steps have been completed Integrated Synchronization is ready to use. You can now start replication and enable tables for replication (for example by using administration client controls from Db2 Analytics Accelerator Studio or from Data Server Manager).#Db2AnalyticsAcceleratorforz/OS#IDAA