You might notice in the logs the following warning message:

[thread:HikariPool-8 housekeeper] [stage:] WARN HikariPool - HikariPool-8 - Thread starvation or clock leap detected (housekeeper delta=2m06s131ms294µs591ns).

It indicates a high load on the engine and could be followed by an error java.lang.OutOfMemoryError: Java heap space.

You should check the CPU and memory usage of this engine.
 
Possible solutions for this issue could be the following:

• Increasing the memory of the engine.
• Rebalancing the jobs running on the engine.
• Dividing a big pipeline into multiple parts.

Here's a reply for that forum post:

Hey! Yeah this warning is pretty common when the engine is under heavy load - nothing to panic about, but definitely worth looking into before it turns into a bigger problem.

That HikariPool housekeeper delta warning basically means the connection pool thread got delayed significantly (over 2 minutes in your case), which usually points to CPU or memory being maxed out on that engine. And if it's followed by OutOfMemoryError: Java heap space, the engine is genuinely running out of memory.

First thing I'd do is check the CPU and memory usage of that engine while the jobs are running - see how close you're getting to the limits.

From there, a few things that usually help:

If memory is the issue, try increasing the heap size for that engine. If you have multiple heavy jobs stacked on the same engine, rebalance them - spread the load across other engines if possible. And if you've got a big pipeline doing a lot in one go, consider breaking it into smaller parts so it's not all hitting memory at once.

Start with the monitoring check first, that'll tell you which fix makes the most sense for your situation. Let us know what you find!

------------------------------
Peter Robert
------------------------------

Thank you for replying and contributing to the community.

I'll add that newer versions of Java support improved performance, garbage collection and memory handling. 

This video describes splitting pipelines into smaller units of work. StreamSets Academy offers a complimentary "how to" course on the topic. 

Since the release of Java 17, we often recommend setting the Java Memory Strategy to Absolute, specifying actual Min/Max heap sizes, and these specific Java Options: 

-XX:+UseAdaptiveSizePolicy -XX:+TieredCompilation -XX:CompileThreshold=5 -XX:+UnlockDiagnosticVMOptions -XX:GCLockerRetryAllocationCount=100 -XX:MaxHeapFreeRatio=30 -XX:MinHeapFreeRatio=10