Db2 for z/OS & Db2ZAI

 View Only
  • 1.  Java vs DB2

    Posted Mon September 16, 2024 10:05 AM

    I got a call from the DB2 admin that my application creates a new connection for each transaction.

    I checked the app running on Kubernetes using the latest JDBC driver jcc 11.5.9 that the connection pool is properly configured. Logging says yes, and also the metrics say yes. From the Java point of view everything looks ok. 70 JDBC connections and millions of usages.

    The zOS admin says he sees millions of connections and even worse millions of threads. The problem with the threads is that they eat up a lot CPU. 

    Now I'm lost. Yes, the app creates 25000 transaction per minute. Is this to much for DB2? I assumed that DB2 also reuses resources/threads. I actually have no idea what happens between the JDBC layer and the database on zOS. Has somebody a link that explains what happens in between? Or an idea what's going on there?



    ------------------------------
    Haug Bürger
    ------------------------------


  • 2.  RE: Java vs DB2

    Posted Tue September 17, 2024 09:54 AM

    What information is the Db2 for z/OS administrator seeing that leads to his conclusion that there are millions of connections and millions of threads on the z/OS side of this application? That would be odd behavior, to say the least. A Java application accessing Db2 for z/OS via TCP/IP communication links typically establishes a reasonable number of connections to the Db2 for z/OS server (70 connections - from the client-side perspective - does not seem at all unreasonable), and these connections tend to persist until the Java application explicitly terminates them (and that typically does not happen often - connections to the Db2 server are established and remain for quite some time and are reused for many transactions). A connection could be terminated by Db2 in case of an "idle thread timeout" situation, but that is generally an unusual occurrence. [If the idle thread timeout value for the Db2 for z/OS is 120 seconds, that would mean that Db2 would terminate the thread - and the associated connection - used by a client-server transaction if that transaction had gotten started and then appeared to Db2 to have gone into a "hung state." In practical terms, this means that the transaction started and did some work and then never committed - lack of the commit means lack of "end of transaction" from Db2's perspective, and that can lead to an idle thread timeout situation.] If millions of "idle thread timeout" situations were occurring on the Db2 server side, I think the Db2 administrator would notice that.

    As for threads, "millions" is similarly unlikely. Db2 for z/OS maintains a pool of threads (called DBATs in Db2 for z/OS lingo - short for database access threads) that are used for network-connected, client-server applications. A given DBAT will normally be used for 200 transactions (500, if the Db2 for z/OS system is at the Version 13 level) before being terminated by Db2 (to free up associated resources - helps to keep the server-side system from getting "gummed up"), so "millions of threads" (meaning millions of DBAT create actions by Db2) would normally require a humungous number of transactions over an extended period of time.

    As for transaction load, 25,000 transactions per minute (a little over 400 per second) is not a big deal for a Db2 for z/OS server. Plenty of Db2 for z/OS systems are processing thousands of client-server transactions per second (potentially N times thousands-per-second if the Db2 system is a so-called data sharing group with N members).  



    ------------------------------
    Robert Catterall
    ------------------------------



  • 3.  RE: Java vs DB2

    Posted Tue September 17, 2024 10:15 AM

    I am not sure if this is the solution. Are you  using con.setAutoCommit(false); ?



    ------------------------------
    Paul Kenney
    ------------------------------



  • 4.  RE: Java vs DB2

    Posted Tue September 17, 2024 11:10 PM
    This is a programming error when using connection pooling.

    Either the connection is not setup correctly to keep the connection pools open for reuse or you are processing them in correctly in the application. This can come about by not handing Db2 errors correctly as well. Have DBA admin scan for errors that could be occurring.
    Also check how you are processing cursors. If not correctly closing them this will prevent a pooled connection from being reuse.

    Also how you are calling for a connect. You be requesting a new connection for every process instead of connect reuse.

    Let’s start by reviewing your connection settings.


    Sent from my iPhone




  • 5.  RE: Java vs DB2

    Posted Wed September 18, 2024 10:18 AM

    That's exactly what our admin said, but our debug logs log the creation of connections with object reference. And the logs tell a different story, last try I got 63 connection objects logged. And that's exactly what the metrics say. I had several code reviews and log analysis on the Java side that say everything is ok on Java side. How can I create millions of connections without anything being logged? Both the logs and the metrics tell the story that everything is fine, the count of usage, the count of connections allocated....
    On Java side there are no errors. I use Spring repositorys with generated finder methods, no explicit cursor handling. There is a Hikari connection pool used which provides the metrics. All pretty standard which I used for several databases where I could see the DB side, there I could see the number of connections. I uses autoCommit default which is true.
    For me it looks like the threads are not being reused. Since I don't have explicit cursors, are there implicit cursors used for queries? Are there any other situations where reusage is forbidden? How about prepared statements? Maybe the jcc jdbc driver requires something special? 



    ------------------------------
    Haug Bürger
    ------------------------------



  • 6.  RE: Java vs DB2

    Posted Wed September 18, 2024 10:45 AM

    Again, I'd like to know what the Db2 for z/OS-side administrator is citing to support the contention that the Java application is driving millions of connections to Db2 and using millions of threads. The reason I ask: regarding threads, for example, sometimes a Db2 monitor report will indicate that a certain workload utilized X number of "threads," but that is a technically incorrect use of the Db2 term "thread" by the Db2 monitor - the number X is not threads, it's Db2 accounting trace records. There is often one Db2 accounting trace record generated per Db2 client-server transaction; thus, there can be a great many Db2 accounting trace records generated for a high-volume client-server workload, and when a Db2 monitor incorrectly labels these as "threads" that can lead to the impression that there are a great many threads being used, when in fact the high "threads" number just indicates a large volume of Db2 accounting trace records generated by a large volume of client-server transactions that are REUSING Db2 DBATs (i.e., DDF threads - the kind of Db2 thread used for client-server applications).

    My point: sometimes server-side data reported by a Db2 monitor can be misleading because fields in a Db2 monitor online display or in a Db2 monitor-generated accounting or statistics report are not correctly labeled. Knowing the source of the Db2 administrator's "millions of connections and millions of threads" contention would help to determine if this is a real issue or if it is related to figures not correctly labeled by a Db2 monitor.



    ------------------------------
    Robert Catterall
    ------------------------------



  • 7.  RE: Java vs DB2

    Posted Wed September 18, 2024 10:03 PM
    There about a lot of DB2 setting that you can make in the db2 Java driver.  If I get something I will those that could pertain to your situation.  
    The autocommit is interesting so I assume spring is using an optimistic locking strategy.  If that is the case you will want to use one of the db2 Java options to set the isolation to UR.  


    Sent from my iPhone





  • 8.  RE: Java vs DB2

    Posted Wed September 18, 2024 10:08 PM
    One question to ask the DBA.   Is the DDF that you connecting to using High performance DBats.  

    If so that to can cause similar issues as DBat should only be used with only programs that are designed for it.   Turning DBats on for all DDF transactions is a mistake.  

    Sent from my iPhone





  • 9.  RE: Java vs DB2

    Posted Thu September 19, 2024 05:00 AM

    For what I have seen using some metrics with Db2 AI for z/OS, the metrics that affect threads and stop pooling are locators, high performance DBATs, held cursors, packages bound with KEEPDYNAMIC. 

    This last effect is described in the documentation: 

    "If you specify KEEPDYNAMIC(YES), DDF server threads that are used to execute KEEPDYNAMIC(YES) packages will remain active. Active DDF server threads are subject to idle thread timeouts."

    https://www.ibm.com/docs/en/db2-for-zos/12?topic=services-keepdynamic

    Your DBA should have a look to all those. 



    ------------------------------
    Diego Cardalliaguet
    IBM Data&AI zStack Leader for EMEA
    IBM
    ------------------------------