Originally posted by: karthikpc

Hi all,

We are trying to process more than 1 GB files using WTX on Z/OS. But we keep gettig Memory allocation errors as below. If anybody knows how to process( Map settings/ JCL changes / any other considerations) > 1Gb files on Z/OS, please let me know about the same.
For > 1GB file:

---

Error message:
Memory Allocation Failed 131072 ./s_paging.c 1852
Mapping return code 2, severity SEVERE:
Not enough memory to execute map
Job step return code is 16
Elapsed CPU time 2.13 seconds
Virtual storage used by engine 0 bytes

---

Less then 1 GB files, we are able to process them with out any memory related issues.
Please let me know if anybody manage to process >1GB files on Z/OS using WTX.

Thanks&Regards,
Karthik
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: repanzer

I used to run under Windows and now I'm Unix, so I can only give you my experience, but I've hit this.

While under Windows, I never had memory issues, probably because the size of the files were never that big.

In my current shop (UNIX-HP) WTX was set up before I got here, and we will get 250 meg files. After hitting a memory error and in talking to IBM, under Unix, each system, or msl (if they are split and not one compound system) is the equivilent of a Unix process (with process ID and everything). According to IBM, there is a 960K memory limit IN WEBSPHERE TRANSFORMATION EXTENDER, NOT IN UNIX, where any one process (msl) can NOT GRAB more then 960K of RAM, or if fails for memory allocation.

If we had requested the "MAGIC" build of the launcher code, you can double that limit, but we were not aware of that at the time.

So what happens is; you start the system, it's at 5k RAm. A 5 meg file is dropped to that system, (IN UNIX), the system grabs 9 megs of RAM to process the file...when it is done processing that file, IT DOES NOT RELEASE THE RAM. It holds it until you "bounce" the process (bring down and back up). So, if you had 10 megs RAM, and one sysetm grab 8 megs of that RAM and never lets it go, the other system only has 2 megs available from the OS.

I do not know (and have no idea) if this is the same on Z/OS.

The largest files we process are INVOIC's (EDIFACT invoice), and we just started using Burst mode, which drove the memory usage down by over 50%. Intead of reading all 20 megs of an input file into memory and validating like "integral/s" (the default setting), it read only one "group" at a time, like one INVOIC, so you can have a 10 GIG file of a million transaction, but if the largest of those transactions is only 500K, the map should only need the resource to process that 500K at one time, and not the resources to process 10 gigs.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: Rich_McFate

It is not accurate to say that Launcher does not release memory after a map has run. It does.

I have heard this from several customers over the years -- usually based on what they see in Management Console. It is accurate to say that the memory allocation in the Management Console on unix does not come down after map runs. What you see in the Management Console on unix is a highwater mark -- i.e. the highest memory allocation point since the Launcher was started.

Curiously, if the Launcher is running on Windows, you can see the memory allocated to the process rise and fall as maps enter and leave. I've never understood why the reporting behavior is different on the two platforms.

In any event, I had the opportunity some years ago to spend several days investigating this with a large bank in France. They were also running on hp-ux (and had the 'magic' version)which gave them something like 1.8M of available memory for the process.

What we discovered is that we could consistently crash the Launcher if we put a large file through twice in succession.

It turns out that the problem is heap fragmentation. The map we were using used some adapters. Somehow, one of the adapter threads was grabbing some memory. As far as we could tell, the thread involved was related to preserving an adapter connection between map runs, so it remained unavailable after the map finished.

We also noticed that the customer's map was using the RUN() function with HANDLEIN(Package(big_honking_file)). When you do that, WTX must find a contiguous area of memory for the big_honking_file object. The memory area that the adapter had grabbed was near enough to the middle of the heap that there wasn't a contiguous area of memory big enough after the map ran the first time.

repanzer is right that burst mode is helpful. It cuts down on the size of the big_honking_file object and greatly reduces the heap fragmentation.

Heap fragmentation is not relevant here, unless the map that abends is running in the Launcher on z/OS.

The behavior of the Package function over the domain of large data objects might be relevant, however, as is the suggestion about burst mode.

Rich
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: karthikpc

Hi Repanzer,

I think i have the same proble what you had on UNIX-HP. I am able to process upto 900MB files. But if it reaches to 1 GB file, i am having memory allocation problem.

---

Error message i get when i execute a map for 1 GB files:
Write error
EDC5167I Access to the UNIX System Services version of the C RTL is denied.
EDC5167I Access to the UNIX System Services version of the C RTL is denied.
stream 2902C318, buffer 2A41A028, size 131072, sysname DD:SYSTMP03
Map execution terminated from ./mvs.c 4482

• End of IBM Websphere Transformation Extender Execution Log for Map XML1SUB1

---

My File Structure is as follows.

FILE
....XML-HDR
....XML-BATCH-HDR(s)
....XML-TRNS-REC(S)

In my input card, I tried with Intergral and BURST modes. But When i use BURST mode, WTX still trying to read complete file instead of reading it as BURST.
Note: I tried wth these settings:
FetchAs > BURST > S / FetchAs > BURST > 1

If i want to burst every batch and perform the conversion, do i need to change the above settings or is there a different way to do the same?

Please let me know your views on the smae.

Regards,
Karthik
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: cmai

Hi Karthik,

if you see this:
> Write error
> EDC5167I Access to the UNIX System Services version of the C RTL is denied.
> EDC5167I Access to the UNIX System Services version of the C RTL is denied.
> stream 2902C318, buffer 2A41A028, size 131072, sysname DD:SYSTMP03

you need a larger allocation for one of your workfiles (SYSTMP03 in this case).
I would expect that you see a 'B37' in your JESYSMSG.

reards
Carsten
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: karthikpc

hi Cmai,

We incresed work file size as below. But we still have the same problem.

//SYSTMP01 DD UNIT=VIO,SPACE=(CYL,(4000,10000)),
// RECFM=FBS,LRECL=32760
//SYSTMP02 DD UNIT=VIO,SPACE=(CYL,(4000,10000)),
// RECFM=FBS,LRECL=32760
//SYSTMP03 DD UNIT=VIO,SPACE=(CYL,(4000,10000)),
// RECFM=FBS,LRECL=32760
//SYSTMP04 DD UNIT=VIO,SPACE=(CYL,(4000,10000)),
// RECFM=FBS,LRECL=32760
...........etc

Can you please let me know if any changes would be required for the above static temp files?

Regards,
karthik
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: cmai

Hi Karthik,

depending on your z/OS configuration there is a limit set how many space can be allocated for VIO by each job.
You have to talk with your sysprog or run a test with 'real' data set instead of VIO.

regards
Carsten
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: karthikpc

Hi ALl,

Thanks a lot for all the Inputs or suggestions you guys provided out here. Finally It worked for me.

Thanks&Regards,
Karthik
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: denisdeloeil

Hi karthikpc,

We have the same problem on a file of similar size to convert to z / OS.
Could you explain the solution that solved your problem please?

Thank you
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: karthikpc

Hi Denisdeloeil,

Sorry for the late. Its been a while that i looked at developerworks.

If you are running WTX maps on Z/OS, then there is a limitation of 900 MB if you are using your input card in Integral Mode. To process more than 900MB, you wil have to use BURST MOde on your input card and also to make your Input card in BURST mode, you will need a bit of customization to your Input type tree.

Let me know if you need more details. If we go for BURST mode, then we are in a position to process 2GB files on mainframe. More than 2GB can not be processed on mainframe at this point in time using WTX 8.1. But i have heard that 8.3 can process up to 4GB in burst mode. However, Integral mode on input card will be able to process 900 MB only.

Regards,
Karthik P C
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: Rich_McFate

Is this running in the command server via JCL or under Launcher?

If it's JCL, are you confident your REGION is big enough?

Do you have workfiles set in memory?

If not, are you using VIO for workfiles?
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: karthikpc

Hi Rich,
1) Yes , we are executing in the command server via JCL

Ex: To create work files on local memory.( this gives error message as below for > 1GB files:
Write error
EDC5167I Access to the UNIX System Services version of the C RTL is denied.
EDC5167I Access to the UNIX System Services version of the C RTL is denied.
stream 2902C318, buffer 2A41A028, size 131072, sysname DD:SYSTMP03
Map execution terminated from ./mvs.c 4482

• End of IBM Websphere Transformation Extender Execution Log for Map XML1SUB1)

//CMDLINE DD *
XML1SUB1 /I1 MAININP1 /O1 MAINOUTO /O2 MAINOTT1
/O3 MAINOTT2
/VX0D,X15 MAININP1 /VX0D,X15 MAINOUTO
/VX0D,X15 MAINOTT1 /VX0D,X15 MAINOTT2
/*

EX2: To create work files on system memory itself ( this gives hiperspace exhausted for more than 1 GB file)

//CMDLINE DD *
XML1SUB1 /I1 MAININP1 /O1 MAINOUTO /O2 MAINOTT1
/O3 MAINOTT2
/VX0D,X15 MAININP1 /VX0D,X15 MAINOUTO
/VX0D,X15 MAINOTT1 /VX0D,X15 MAINOTT2 /WH
/*
2) If it's JCL, are you confident your REGION is big enough?

Answer is : Yes, we tried with REGION = 900 M / REGION = 2047M / REGION = 0M / With out REGION. But error is the same.

3) Do you have workfiles set in memory?

No. Workfile set in File. Becase of performance issues. IBM says, Work fike set in file is better interms of handling Large Files. ( we tried with Set in Memory as well. but error is the same)
4)If not, are you using VIO for workfiles?

Yes, We are using VIO for work files.
Please let me know if you need more details on this.

Regards,
Karthik
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender