App Connect

Are you sure your flow doesn't have a memory leak problem?

Hi @Francois Brandelik, I am not exactly sure. We are using compute nodes and custom java code. We also have a java jdbc connection pool implementation. Could this be due to network issues as well? There is no specific pattern of occurance for this problem. We noticed the following messages in our logs:

Mar 22 03:03:34 app-connect01 ACE[1279752]: IBM App Connect Enterprise v120126 (PRD01BRK.eg_TMFBService) [Thread 2258486] (Msg 1/3) BIP3120E: Exception condition detected on input node 'gen.TMFBService.HTTP Input'.
Mar 22 03:03:34 app-connect01 ACE[1279752]: IBM App Connect Enterprise v120126 (PRD01BRK.eg_TMFBService) [Thread 2258486] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'gen.TMFBService.HTTP Reply'.
Mar 22 03:03:34 app-connect01 ACE[1279752]: IBM App Connect Enterprise v120126 (PRD01BRK.eg_TMFBService) [Thread 2258486] (Msg 3/3) BIP3742E: An attempt was made to use a EVHT message ReplyIdentifier after it had already been used.

Produce dumps (java dumps) and analyze them. You should also make sure that you release the java used native memory before going onto the next message...

Hi @Francois Brandelik,I managed to get the dump using the mqsicommand:

mqsichangeproperties TDEVBK -e eg_TMFBService -o ComIbmInternalSupportManager/Java/JVM -n dump -v all

I used the following documentation: https://www.ibm.com/support/pages/generating-java-diagnostic-dumps-app-connect-enterprise-or-ibm-integration-bus.

Also, i would like to mention that we haven't seen OutOfMemoryException in log files.

Hi Irfan!

I have seen this error before (BIP3742E: An attempt was made to use a EVHT message ReplyIdentifier after it had already been used.) And this happens if you try to respond twice for one call. Typically the culprit is a PROPAGATE TO 'out' followed by RETURN TRUE f.ex. 

Check your ESQL and Java nodes and make sure that the code doesn't propagate AND RETURN TRUE to the final reply node. 

This is also confirmed by looking up the error code which has this explanation:
"BIP3742E: An attempt was made to use a insert2 message insert3 after it had already been used.
A insert2 message insert3 may only be used once to send a single reply (or a single async response). The insert2 message insert3 was insert1.
Determine the cause of the error and correct it. Subsequent error messages may contain more information."

Hope that helps.

Cheers,

Lars

Hi @Francois Brandelik,

We added a call to a subflow to the main message flow of the RestAPI project (despite a warning message from the toolkit that the messageflow should not be changed manually). The subflow includes a compute node that calls a java method to get the hostname of the machine and set it as one of the http response headers. Could this be causing problems? I have attached screenshots of these as well.

The other reason you'd see this behavior is in case of a timeout...

Now you'd have to look at which part of the flow times out....

What method are you using in Java to get the hostname?

Thanks for your reply @Francois Brandelik, We are using the following java code to get hostname. When timeout occurs, it appears to be the reply node that doesnt respond. But i am not 100% sure. How can we isolate the node/ code that might be causing this?

public static String getHostname() {
    try {
      InetAddress ip = InetAddress.getLocalHost();
      return ip.getHostName();
    } catch (Exception e) {
      return "Error: Unable to retrieve hostname.";
    } 
  }

Hi!
This is something you should either try to debug or, if debugging is not possible, try to add some log statements to see where the messages goes? It seems one of the paths it can take doesn't reach the reply node. 

Hi Irfan,

Please not that your method to get a local IP is not reliable when you have multiple interfaces/IP addresses on the same server.

If you want to supply the local IP used as source for a specific destination, you should start with a DatagramSocket to your target host and use that to get the local address.

Note that the local address should then match what you have in your routing table.

Can you please explain what you are trying to achieve when putting the hostname on the response header?

Hope it helps

Hi @Francois Brandelik, We wanted a way to know which appconnect node served the request at the api gateway level. Thats the reason behind this custom response header .

Hi Irfan
Did you find a solution?

Additional question, you did not change the generated message flow, have you?

You mention the ACE logs, but you have no explicit logging in your screenshots. You mean the activity log then?

The fact that it works after restarting does seem to indicate en issue like a memory leak or open ipprocs or something else that builds over time.

The 3742 error indicates you send multiple replies, but that is not clear from your screenshots. Can  you check the log timestamps, is 3742 coming from the reply your API gives or from a bit later, after the configured timeout. 
How heavy is that flow taxed? You mention restarting the nodes, so I assume you are running ACE Nodes with multiple integration servers. Are you using node wide listener or embedded listener?

You also mention JDBC, but not used by this flow then?

Hi @Irfan Rashid

Did you get it working?

Hi @Matthias Blomme,

We are still trying to find the root cause. As a temporary fix, we restart the appconnect nodes after a week or so.