Originally posted by: SystemAdmin

In the beginning of the process of the map, it always validate the input data file with the input card type tree. Is there a way to turn this option or function off?
I have a large file that the output is simple but the process of validating the input file took a long time. (more than 1 hr).

Please if you don't mind, give me your suggestion.

Thanks.

EDI-Hipaa hat!!
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: repanzer1

I don't believe you can. There are options in the map settings that let you adjust whether to ignore some types of errors, but it still validates the data. Something’s you can do to try to get the map to execute quicker is to alter the page settings under memory.

I'm assuming you do not have the trace file on, or the audit log creation with audit log settings. A map with the trace file on takes much longer to process.

---

There are 10 types of people in this world. Those who understand binary and those who don't.
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: repanzer1

The only other possible option I know is to make the type tree less complex. If there is any data in the input, like a group of ten fields that you do not map in you output, you can make those 10 fields a text blob. Anything you can do that makes validation less stressful will speed up the map execution.

---

There are 10 types of people in this world. Those who understand binary and those who don't.
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: SystemAdmin

Thanks for your tip.

No, I don't have trace on, I don't have audit on either.
The type tree is HIPAA_x12 type tree from IBM. I deleted all the transactions that I don't need to make the type tree smaller. It helped cut down about 1 hr.
Please explain what you mean about "Something’s you can do to try to get the map to execute quicker is to alter the page settings under memory."
How do I set that under memory?

Do you know about "burst" option in the "input card/fetch as"? I don't know the different between "burst" and "integral".

Best regards,

EDIx12HIPAA hat
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: repanzer1

Obviously all these topics are in the documentation and reading it can give you a much better idea than what I can post here. If you go into map settings, you can change the page count and page size of the memory when the map runs. Depending on your input, changing it may help the map run more efficiently. I’m not entirely sure what goes on behind the scenes with the settings, I only have an idea, but try messing with it and see what happens.

Burst mode is usually used when processing very, very large files. Instead of the map reading in the entire input file like it normally does, it will process “bursts” instead, reading a portion of the data set by the programmer and processing it, then reading the next burst, and so on. If you have an interchange of three 835’s, normally the map reads the three transactions then processes them, but you can have it read in one, process it, then read the next one, etc. I’m not sure how much this would effect processing time though, if at all.

Again, all of this is in the documentation and I highly recommend reading it….all of it. There may be many aspects of WTX you’re missing because you simply don’t know it exists. I have personally read most of the documents more then once. If I see something in it I don’t fully understand, I’ll code it to see what happens, like if I don’t understand exactly what one of the functions does, I’ll build a simple type tree/map and use the function until I understand it. You may be surprised by what WTX can handle, and if you’re not aware of setting the page count and size in the map settings, it’s obvious there is tons more functionality you can learn. There’s an entire manual for “Performance Recommendations” with the description;

“Providing tips and recommendations to improve performance of the WebSphere Transformation Extender products. Strategies are presented to leverage the flexibility and power of the WebSphere Transformation Extender products to achieve reasonable performance objectives.”

---

There are 10 types of people in this world. Those who understand binary and those who don't.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: repanzer1

Here is an excerpt from the performance manual;

Reusing work files;

Work files contain metadata about the various type objects found in each input data file. If the data content is static, this metadata can be reused in subsequent runs. Reusing work files allows the transformation engine to bypass the validation step for these sources. Reducing the number of steps results in a shorter overall execution time.
If I had not read the documentation, there is no way I would have ever known this, as well as so many more topics.

---

There are 10 types of people in this world. Those who understand binary and those who don't.
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

Thanks for your help. I do create small maps to test functions or to learn certain functions just because they don't have good explanation in the manual. There are certain topics that I still not clear.
It's very nice that you posted your suggestion.
Best regards,
edix12hipaa hat
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: FormerBuckeye

I believe reusing a work file only applies if you are re-translating the same data in the same file. (I use this method when translating a single source file to multiple outbound formats for different trading partners.) I don't believe it applies to multiple files of the same format with different data. Frankly, even it that worked, I don't trust my internal or external trading partners not to have made some change that hoses their data.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: jvanboga

Curious, how big is big? For us a large hipaa file is 75 - 100 mb. We can read, map and load a 100 mb hipaa transaction into 13 oracle tables in less than 10 minutes. That includes post mapping edit processes.

You've already stated that you've turned off map logs and traces so...

Performance enhancers can depend on the environment. For example, if you allow Windows to manage disk memory on the fly performance can be negatively impacted. I have found that our maps run faster when you preallocate space.

They can also be impacted by type tree design. Unfortunately, with hipaa files you want a certain level of validation going on when you read the file or you run the risk of having data integrity problems. We have historically used the type-1 hipaa trees which, for the most part only validate structure. They do not, for example check the punctuation of an SSN.

As stated above, using burst mode can really help giving you a speed boost if the tree is designed properly. If not, burst will add little to performance. Historically the hipaa pack type trees have been pretty well designed.

Map design/structure will affect your performance as well. Poorly structured maps can really suck up resources and negatively affect your processing time. You can prove this by designing a map two different ways, run each and watch what happens to your temp files.

Playing with pagesize/count settings can help boost performance as well. I've found there isn't a single best way to configure this. You have to play with it.

Usually, I don't see a benefit between 'create' and 'reuse' but others seem to.

Are you reading any DB data in your map?
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange