IBM Sterling Transformation Extender

Sterling Transformation Extender

Come for answers, stay for best practices. All we're missing is you.

 View Only

  • 1.  Large JSON processing error in ITX 10

    Posted Thu January 06, 2022 02:36 AM
    Hi Team

    I have a map which converts JSON to Flat file.
    My source payload is around 1.5GB and once it converted to flat flie size is around 3.1GB
    Map setting 
    Workspace Memory
    Page count 1024
    Page size 64
    On success delete for the source card.
    Integral Mode
    Trace Off
    Audit On

    also modified dtx.ini
    Streammaxlimit = 500

    streammaxdirectory = /itxa/tmp directory 

    operating system is linux
    runtime engine is SPE via Sterling B2Bi

    it works fine on every restart of the ITX server. but after some time when the map triggers then itx server is going to hanging state.
    no logs ,  in logs directory it created heading of the audit log with .tmp file extension and sterling dashboard is not accessible there after.
    Also it is copying the source payload into /itxa/tmp directory twice and occupying the disk space of tmp directory. (mpi* filename with source payload are copied)

    what could be the best tunning for this map. and i want to understand how the memory management is done during the runtime.

    Thanks
    Siva Kumar

    ------------------------------
    siva kumar
    ------------------------------

    #DataExchange
    #IBMSterlingTransformationExtender


  • 2.  RE: Large JSON processing error in ITX 10

    Posted Thu January 06, 2022 02:48 AM
    Also I have around 300gb in tmp directory.

    Thanks
    Siva Kumar

    ------------------------------
    siva kumar
    ------------------------------

    Original Message


  • 3.  RE: Large JSON processing error in ITX 10

    Posted Thu January 06, 2022 02:38 PM
    For Large Files, I usually do not use Memory, I use File as workspace.
    I don't think it will impact performance much unless you have many concurrent maps with large input hitting the same disk volume causing I/O contention.

    Also I would change the Page size so it could reduce the number of times to get blocks of memory.
    Start with 256 and work your way up to find the optimum size.  This may improve performance.

    Regardless, do one change at a time.

    Your setting:
    Workspace Memory
    Page count 1024
    Page size 64

    On the topic of ITX Server, there is no such thing as ITX Server when running SPE service in B2Bi.
    Maps are invoked via B2Bi Business Process using local Java ITX API call.   This is why there are ITX jars defined in the dynamicclasspath.cfg properties file.  The map runs in B2Bi's JVM.

    If you are referring to the SPE UI (it uses Liberty Server behind the scene), it's not used during runtime.  As a matter, you can shut it down and the map will still run in B2Bi.

    ------------------------------
    Rex Chan
    ------------------------------

    Original Message


  • 4.  RE: Large JSON processing error in ITX 10

    Posted Mon January 10, 2022 11:33 AM
    Hi Siva Kumar -

    Here are my thoughts about your post and map tuning in general.

    First, I would start with some basic questions such as:
    - How often will the map be executed?
    - How important is performance?
    - Will there be competition for resources?

    And then some application specific questions:
    - What is the nature of the inputs and outputs?
      - Is there a typical size or size range?
      - Is the size consistent or variable?
      - Are there looping structures or hierarchy?
    - How complex are the mapping requirements?
      - Are inputs being combined?
      - Are there multiple outputs?
      - Are any of the outputs used as inputs for subsequent outputs?
      - What are the expected commit/rollback requirements?

    You've said that the input to your map is formatted as JSON, your output is a flat file, and both are very large. That's not a lot to go on but it does lead me to suggest that you look for ways to break the input into smaller chunks that can be used to generate chunks of output. Going down that path will require analysis and additional development effort up-front but should allow for better performance and resource utilization.

    If you'd like to provide more information about your specific requirements either in this forum or via e-mail, I should be able to go a bit further with recommendations.

    Regards,
    Kristen Stenberg
    Senior Integration Consultant at CoEnterprise
    P 973-769-6950 | E kristen.stenberg@coenterprise.com

    ------------------------------
    Kristen Stenberg
    ------------------------------

    Original Message


  • 5.  RE: Large JSON processing error in ITX 10

    Posted Mon January 10, 2022 11:57 AM
    Hi Siva Kumar -

    Here are some of my thoughts about your map tuning question.

    First, I would start with some basic questions such as:
    - How often will the map be executed?
    - How important is performance?
    - Will there be competition for resources?

    And then some application specific questions:
    - What is the nature of the inputs and outputs?
    - Is there a typical size or size range?
    - Is the size consistent or variable?
    - Are there looping structures or hierarchy?
    - How complex are the mapping requirements?
    - Are inputs being combined?
    - Are there multiple outputs?
    - Are any of the outputs used as inputs for subsequent outputs?
    - What are the expected commit/rollback requirements?

    You've said that the input to your map is formatted as JSON, your output is a flat file, and both are very large. That's not a lot to go on but it leads me to suggest that you look for ways to break the input into smaller chunks that can be used to generate chunks of output. Going down that path will require analysis and additional development effort up-front but should allow for better performance and resource utilization.

    If you would like to provide more details about your requirements via this thread or in an email then I might be able to provide more specific recommendations.

    Regards,
    Kristen Stenberg
    Senior Integration Consultant, CoEnterprise
    P 973-769-6950 | E kristen.stenberg@coenterprise.com

    ------------------------------
    Kristen Stenberg
    ------------------------------

    Original Message


Related Content