Hello
@Matthias Blomme,
It's not easy to construct general guidance that applies in all cases, unfortunately, as a lot depends on what the flows are actually doing.
For example, a flow that is CPU-intensive (lots of parsing, mapping, etc) is unlikely to benefit from being given many more threads than the number of CPUs available: it will simply waste time in context switching and make less overall progress than if it had a more sensible number of threads. This is especially true for larger messages due to the limited size of processor L1/L2 caches, as excessive context switching will force the CPU to flush and reload cache lines a lot, slowing the system down in a way that is hard to capture with most monitoring tools. We've seen this sort of situation with customers, and the giveaway is normally the number of context switches per second.
On the other hand, a flow that spends a lot of time waiting for remote systems (synchronous HTTP requests, database stored procedures, etc) is a much better candidate for using lots of threads, and in fact that may be the only way to get sufficient throughput. If an HTTP service takes ten seconds to respond, then ten threads would be needed to maintain a message rate of one message per second (assuming the service itself handles simultaneous requests). This was quite a common scenario in the early days of the HTTPRequest node before the HTTPAsync nodes were added, and can still happen in situations where an async model isn't appropriate (transactional concerns, non-HTTP services, etc). There's still context switching and cache reloading, but a lot of that would happen anyway if a flow blocks for ten seconds.
Different CPU types (x86_64, Power, etc) also behave slightly differently in this sort of area, with some working better than others at higher thread loads. There aren't that many general rules, other than
- Using async nodes and avoiding blocking the flows will allow higher performance with fewer threads, saving memory and context switching, but this is not always appropriate for all flows.
- Simply adding more threads to "make things go faster" (which we've seen customers try to do) doesn't generally work, as more thought is usually needed than just changing one variable in the system.
Hope that makes some sense! You've probably already thought of a lot of these things before anyway . . .
Edit: the close_wait issue sounds like it might be a product defect, though more details might be needed to be sure.
------------------------------
Trevor Dolby
IBM Expert Labs
------------------------------
Original Message:
Sent: Tue November 08, 2022 01:17 AM
From: Matthias Blomme
Subject: Maximum thread for an integration server
Hi Lars
We have one big integration server with a lot of applications and thus a lot of threads.
There are cpu spikes and we see http issues occuring (close_wait) from the moment an endpoint starts behaving slower then usual.
During investigation I was wondering if there is a limit on the threads in an IS. Logic dictates spreading the load out over multiple IS is more performant for http, i/o, cpu contention, debugging, ... But I was looking for some guidance from IBM here.
------------------------------
Regards
Matthias Blomme
Original Message:
Sent: Mon November 07, 2022 08:13 AM
From: Lars Dehli
Subject: Maximum thread for an integration server
Hi Matthias.
Do you have a specific goal you want to achieve or is this a theroretical duscussion? Normally, tuning number of threads is highly specific to what the flows does (what kind of resources does it access and so on) in combination with performance requirements.
------------------------------
Lars Dehli
Original Message:
Sent: Thu November 03, 2022 03:57 AM
From: Matthias Blomme
Subject: Maximum thread for an integration server
Hi Gabriel
If there is no actual limit, I assume there is a best practice? Having that many flows orchestrated by a single thread can't be very efficient?
Having multiple integraton servers, and thus multiple process, should be better for the server cpu to handle cpu processing time.
3000 active threads seems to be a lot!
Is there any advice on it?
------------------------------
Regards
Matthias Blomme
Original Message:
Sent: Tue November 01, 2022 03:57 PM
From: Gabriel Marte Blanco
Subject: Maximum thread for an integration server
Matthias,
I escalated your question and one of our support engineers just replied to me. According to the support engineer:
"We have no thread limit. We have a limit on the number of instances of a single flow, which is 256. However, you could have multiple flows with 256 instances. once had a client with 3000."
So it seems that the limit is on the instances of a single flow, which you can read this documentation to learn how to update it.
------------------------------
Gabriel Marte Blanco
Austin TX
Original Message:
Sent: Fri October 28, 2022 04:22 AM
From: Matthias Blomme
Subject: Maximum thread for an integration server
Several years back there was a limit on the number of threads in one integration server of 256.
https://www.ibm.com/mysupport/s/question/0D50z000062k9hwCAA/max-message-flows-per-integration-servers-?language=en_US
Is this limit still in place? Or is there a guideline on the the number of threads per integration server?
------------------------------
Regards
Matthias Blomme
------------------------------