IBM MQ Family

Expand all | Collapse all

IIB HTTP Reply Node and Large Messages

  • 1.  IIB HTTP Reply Node and Large Messages

    Posted 15 days ago

    I have an IIB message flow that looks like this:

    HTTP Input Node -> File Read Node -> HTTP Reply Node.

    (1) HTTP Input Node - flow is started by a user initiating an https request. (2) File Read Node - message flow gets a file from our ftp server (local file directory, file name, sftp host and credentials, and sftp server directory all temporarily hardcoded just for ease in troubleshooting a solution, but in real life this info would get extracted from the http request). (3) HTTP Reply Node - reply to user with contents of file retrieved from the ftp server.

    The problem is this: The message flow causes a "java/lang/OutOfMemoryError" heap dump when the File Read Node reads large files (I've been testing by running multiple instances of the message flow with a 124 MG file).

    Background: This is a legacy ASP/VBScript application that was rewritten in IIB. I've been tasked with fixing the memory error.

    More Info: From reading, I understand why the committed memory gets all used up. I've read all I can on large message processing but because this application's main purpose is to allow users to pick up files from our ftp server via https, I seem to be tied to that HTTP Reply Node and thus the entire file has to be available in the MBMessageAssembly object (and therefore in memory) when the flow hits the "in" terminal of the HTTP Reply Node.

    My question is this: Is there a technique for "streaming" to the HTTP Reply Node? Or, is it possible to remove the HTTP Reply Node altogether and do the SFTP and the HTTP Reply in a Java Compute Node (I can't use a Compute Node/ESQL since we aren't licensed for that)? Or, any other ideas?



    ------------------------------
    Carolyn Poreda
    ------------------------------


  • 2.  RE: IIB HTTP Reply Node and Large Messages

    Posted 14 days ago
      |   view attached
    Hello Carolyn,

    I have created a simple flow that takes an HTTP request, then read a 24mb .exe file (it's all I had with the correct size) and then squirt it out through the HTTP reply node.  I have attached a screenshot of the SoapUI project.  I did not run into memory or any limit issues in IIB.  The response did take 5 seconds :-0
    I am not parsing the message or manipulating it in any way.  Record detection in the file read node is set to 'whole file'.  
    You must be doing something else that is causing the issue as the HTTP reply can handle 24MB.

    Best regards
    Carl

    ------------------------------
    Carl Burger
    ------------------------------



  • 3.  RE: IIB HTTP Reply Node and Large Messages

    Posted 14 days ago
    ​Hi Carl,
    Thanks for your feedback. I blow the JVM when I run multiple instances of a 124 MG file, not one instance of a 24 MG file : ) From my post: "The problem is this: The message flow causes a "java/lang/OutOfMemoryError" heap dump when the File Read Node reads large files (I've been testing by running multiple instances of the message flow with a 124 MG file)."
    So I'm wondering: does the entire file contents have to be in memory in order to send it out through an HTTP Reply Node, or, alternatively, is there a technique where I can stream it to the HTTP Reply Node?

    ------------------------------
    Carolyn Poreda
    ------------------------------