Hi Team

I have a map which converts JSON to Flat file.
My source payload is around 1.5GB and once it converted to flat flie size is around 3.1GB
Map setting 
Workspace Memory
Page count 1024
Page size 64
On success delete for the source card.
Integral Mode
Trace Off
Audit On

also modified dtx.ini
Streammaxlimit = 500

streammaxdirectory = /itxa/tmp directory 

operating system is linux
runtime engine is SPE via Sterling B2Bi

it works fine on every restart of the ITX server. but after some time when the map triggers then itx server is going to hanging state.
no logs ,  in logs directory it created heading of the audit log with .tmp file extension and sterling dashboard is not accessible there after.
Also it is copying the source payload into /itxa/tmp directory twice and occupying the disk space of tmp directory. (mpi* filename with source payload are copied)

what could be the best tunning for this map. and i want to understand how the memory management is done during the runtime.

Thanks
Siva Kumar

------------------------------
siva kumar
------------------------------

#DataExchange
#IBMSterlingTransformationExtender

Also I have around 300gb in tmp directory.

Thanks
Siva Kumar

------------------------------
siva kumar
------------------------------

For Large Files, I usually do not use Memory, I use File as workspace.
I don't think it will impact performance much unless you have many concurrent maps with large input hitting the same disk volume causing I/O contention.

Also I would change the Page size so it could reduce the number of times to get blocks of memory.
Start with 256 and work your way up to find the optimum size.  This may improve performance.

Regardless, do one change at a time.

Your setting:
Workspace Memory
Page count 1024
Page size 64

On the topic of ITX Server, there is no such thing as ITX Server when running SPE service in B2Bi.
Maps are invoked via B2Bi Business Process using local Java ITX API call.   This is why there are ITX jars defined in the dynamicclasspath.cfg properties file.  The map runs in B2Bi's JVM.

If you are referring to the SPE UI (it uses Liberty Server behind the scene), it's not used during runtime.  As a matter, you can shut it down and the map will still run in B2Bi.

------------------------------
Rex Chan
------------------------------

Original Message:
Sent: Thu January 06, 2022 02:47 AM
From: siva kumar
Subject: Large JSON processing error in ITX 10

Also I have around 300gb in tmp directory.

Thanks
Siva Kumar

------------------------------
siva kumar
------------------------------

Hi Siva Kumar -

Here are my thoughts about your post and map tuning in general.

First, I would start with some basic questions such as:
- How often will the map be executed?
- How important is performance?
- Will there be competition for resources?

And then some application specific questions:
- What is the nature of the inputs and outputs?
  - Is there a typical size or size range?
  - Is the size consistent or variable?
  - Are there looping structures or hierarchy?
- How complex are the mapping requirements?
  - Are inputs being combined?
  - Are there multiple outputs?
  - Are any of the outputs used as inputs for subsequent outputs?
  - What are the expected commit/rollback requirements?

You've said that the input to your map is formatted as JSON, your output is a flat file, and both are very large. That's not a lot to go on but it does lead me to suggest that you look for ways to break the input into smaller chunks that can be used to generate chunks of output. Going down that path will require analysis and additional development effort up-front but should allow for better performance and resource utilization.

If you'd like to provide more information about your specific requirements either in this forum or via e-mail, I should be able to go a bit further with recommendations.

Regards,
Kristen Stenberg
Senior Integration Consultant at CoEnterprise
P 973-769-6950 | E kristen.stenberg@coenterprise.com

------------------------------
Kristen Stenberg
------------------------------

Hi Siva Kumar -

Here are some of my thoughts about your map tuning question.

First, I would start with some basic questions such as:
- How often will the map be executed?
- How important is performance?
- Will there be competition for resources?

And then some application specific questions:
- What is the nature of the inputs and outputs?
- Is there a typical size or size range?
- Is the size consistent or variable?
- Are there looping structures or hierarchy?
- How complex are the mapping requirements?
- Are inputs being combined?
- Are there multiple outputs?
- Are any of the outputs used as inputs for subsequent outputs?
- What are the expected commit/rollback requirements?

You've said that the input to your map is formatted as JSON, your output is a flat file, and both are very large. That's not a lot to go on but it leads me to suggest that you look for ways to break the input into smaller chunks that can be used to generate chunks of output. Going down that path will require analysis and additional development effort up-front but should allow for better performance and resource utilization.

If you would like to provide more details about your requirements via this thread or in an email then I might be able to provide more specific recommendations.

Regards,
Kristen Stenberg
Senior Integration Consultant, CoEnterprise
P 973-769-6950 | E kristen.stenberg@coenterprise.com

------------------------------
Kristen Stenberg
------------------------------