By default, Query Monitor uses a built-in encryption method (RC4) for B (between CAE server and CAE agent) and C (between CAE agent and ISPF client) – see diagram above. You can optionally configure AT-TLS for communication B and C if desired. 

Why encrypt the connection between ISPF, agent, and server?

QM can collect dynamic SQL text which can contain sensitive data, e.g., SELECT … FROM table WHERE CREDITCARDNUMBER = ‘…’ or UPDATE table SET NAME = ‘Jørn Thyssen’, ADDRESS = ‘…’ WHERE CUSTNO = ….

QM can also collect host variables for static and dynamic SQL, which will very likely also contain sensitive data.

Hence encryption of connection may be required even though the network traffic is within the LPAR and sysplex.

Other customers have a policy to use strong encryption on anything that opens a port. 

The default built-in encryption method RC4 has known vulnerabilities and is not considered secure in 2023. Most customers require use of TLS v1.2 or TLS v1.3 with known good ciphers. 

Configuring AT-TLS communication between CAE server and CAE agent (indicated with B in the diagram above) is described in detail in the manual: https://www.ibm.com/docs/en/dqmfz/3.4.0?topic=monitor-configuring-tls-security

Configuration of AT-TLS communication for the communication between CAE agent and ISPF client (indicated with C in the diagram above) is currently not documented in the manual.

Furthermore, the CAE server listens on several other ports for internal usage, such as management and discoverability. Setting up AT-TLS communication for these are not documented and is outside the scope of this blog as no sensitive information is transmitted. 

In this blog we will discuss some of the options, lessons learned, and pitfalls. 

Rolling migration

The CAE server supports both AT-TLS and RC4 connections, so it is possible to have a rolling migration LPAR for LPAR. Hence, we strongly recommend enabling AT-TLS on a single LPAR first, and then migrate LPAR by LPAR until all LPARs are done.

Once all LPARs are using AT-TLS for QM CAE communication, then the port with RC4 communication can be turned off (see https://www.ibm.com/docs/en/dqmfz/3.4.0?topic=security-preventing-cae-server-from-opening-rc4-sockets) 

AT-TLS rules - for communication between CAE server and agent only

The manual documents two AT-TLS rules

Rule #1: incoming traffic to the CAE agent from the CAE server
Rule #2: outgoing traffic from the CAE agent to the CAE server

Important: the CAE server is a java program and supports native TLS, so no AT-TLS rules are required for the CAE server side.

This scenario is documented in the manual https://www.ibm.com/docs/en/dqmfz/3.4.0?topic=security-configuring-tls-policy-statements

Notes:

• Please review the TTLSEnvironmentAdvancedParms section. For example, you may want to add TLS v1.3 
• Please review the TTLSCipherParms section. If you add TLS v1.3 you need to add TLS v1.3 supported ciphers; if you remove TLS v1.2 you can remove the TLS v1.2 ciphers. There are various sources on the internet that lists recommended cipher suites, e.g., https://wiki.mozilla.org/Security/Server_Side_TLS 

AT-TLS rules - for communication between CAE server, agent and ISPF

This scenario is not documented in the manual and requires changing rule #1 as well as adding a new rule #3:

Rule #1 (modified!): incoming traffic to the CAE agent from the CAE server and ISPF client.
Rule #2 (unchanged): outgoing traffic from the CAE agent to the CAE server 
Rule #3 (new!): outgoing traffic from the ISPF client

Modified Rule #1:

• Remove the RemoteAddr keyword.
• LocalPortRange must include both agent listener ports and ISPF listener port. Unless those ports are in direct sequence, you must use a LocalPortGroupRef construct.

TTLSRule CaeAgentInboundRule

{

    LocalAddr ALL

    LocalPortGroupRef        CQMCAE-IN@PortGroup 

    Jobname                  $CAE_AGENT_JOB_NAME

    Direction                Inbound

    TTLSGroupActionRef       groupCaeAgentInboundRule

    TTLSEnvironmentActionRef envCaeAgentInboundRule

    TTLSConnectionActionRef  connectionCaeAgentInboundRule

}

PortGroup CQMCAE-IN@PortGroup             

{                                         

  PortRange                               

  {                                       

     Port                    $CAE_ISPF_LISTENER_PORT                           

  }                                       

  PortRange                               

  {                                       

     Port                    $CAE_AGENT_LISTENER_PORTS                    

  }                                       

}  

New rule #3 – outgoing traffic from the ISPF client

TTLSRule CQMISPF-OUT@Rule                      
{                                              
  RemoteAddr               127.0.0.1           
  RemotePortRange          $CAE_ISPF_LISTENER_PORT                  
  Direction                Outbound            
  TTLSGroupActionRef       groupQmISPFOutboundRule
  TTLSEnvironmentActionRef envQmISPFOutboundRule
  TTLSConnectionActionRef  connectionQmISPFOutboundRule
}                                              

• RemoteAddr is 127.0.0.1 (localhost) as the ISPF client only talks to the CAE agent on the same LPAR.
• Instead of new groupQmISPFOutboundRule, envQmISPFOutboundRule, and connectionQmISPFOutboundRule you can reuse the groupCaeAgentOutboundRule, envCaeAgentOutboundRule, and connectionCaeAgentOutboundRule as the settings would be the same in most cases (step 3 in the manual https://www.ibm.com/docs/en/dqmfz/3.4.0?topic=security-configuring-tls-policy-statements) 
  

Keyrings and certificates 

In the most complex case, you will have four keyrings, three certificates:

• Keyring 0 (CAE server keystore and trust store)
  • External server certificate for HTTPS connections
  • Root CA for external server certificate for HTTPS connections
  • Internal TLS certificate for CAE server
    Note: you may reuse the External server certificate for HTTPS connections for this purpose.
  • Root CA for internal TLS certificate for CAE server
  • Root CA for the internal TLS certificate for incoming CAE agent communication
• Keyring 1 (for AT-TLS Rule #1)
  • Internal TLS certificate for incoming CAE agent communication
  • Root CA for this TLS certificate
• Keyring 2 (for AT-TLS Rule #2)
  • Root CA for internal TLS certificate for CAE server
• Keyring 3 (for AT-TLS Rule #3)
  • Root CA for internal TLS certificate for CAE agent

Notes: 

• Tip: instead of an explicit keyring with the correct certificate authorities for Rule #2 and Rule #3, you can use an implicit keyring of all certificate authorities trusted in your ESM:
  Keyring *AUTH*/* 
• For simplicity use the same certificate for all QM TLS communication (external web and internal). In that case, you can use the same keyring for all the components. 

Configuration changes

CAE server

Add the following two parameters to the CAE server parameters:

CQM_TLS_CAE_AGNT_LSTNR_PORT=<value>
CQM_TLS_KEY_ALIAS=<value>

The first parameter is the new port for AT-TLS communication between the CAE server and agent. Make sure this port has been reserved if required. 

If you do not provide the TLS key alias for the internal communication with the CAE agent, then the QM server will issue a warning, and use the default certificate in the keyring.

Restart the server after changing the configuration.

Note that this configuration change can be done without the AT-TLS policies in place: the CAE server will continue to accept RC4 communication on the normal agent listener port.

Once all agents have been migrated to AT-TLS, you can turn off the RC4 port with:
RC4_PROTOCOL_ENABLED=false

CAE agent

Update (or add) the following parameters:

SERVER_PORT(<new value>)
ATTLS_ENABLED(Y)

The SERVER_PORT should now reference the CAE server TLS enabled port (CQM_TLS_CAE_AGNT_LSTNR_PORT) instead of the RC4 enabled port (CQM_CAE_AGENT_LISTENER_PORT).

ATTLS_ENABLED should be Y.

Note that you will likely have to add the ATTLS_ENABLED keyword as it is not present in the sample member generated by the Tools Customizer.

Note that the CAE agent config can be rolled out LPAR-by-LPAR for testing purposes. 

Restart the agent after changing the configuration. Note that this must be coordinated with the activation of the AT-TLS policies on that LPAR. 

Troubleshooting

Based on support cases and other customer engagements the typical mistakes made are:

• Enabling AT-TLS for any incoming traffic to the CAE server – this is not needed as the CAE server is a java application with native TLS support.
• Certificate related issues – go for the simple setup with a single keyring shared by the CAE server and all AT-TLS rules if allowed. If you have a more complex setup, make sure that the right certificates and root CAs are in the right keyrings.
• Use the “Trace 255” in the TTLSEnvironmentAction to add tracing ; see https://www.ibm.com/docs/en/zos/3.1.0?topic=statements-ttlsenvironmentaction-statement
• Make sure you make the needed configuration changes described in the previous section before enabling the AT-TLS policies.
• Carefully review all the rules & ports 
• If authorized, you can use the z/OS Unix command “pasearch” to display active AT-TLS policies.
• Use tools like openssl and sslyze to check TLS setup