IBM Sterling Transformation Extender

 

If you are reading this, then you most likely are familiar with the finding errors processing input by using the ITX trace file.  The trace file, or mtr file, is a very powerful debugging tool that mappers have been using since Mercator was first released, to determine parsing failures.   Failure may be due to the customer sending bad data, or just maybe, a mistake was made in the type tree that was built to describe the data.

 

For those familiar with trace, you know that it is an extremely verbose description of the steps the ITX parser takes in its journey to tokenize the data.  Regardless of the complexity of the type tree, the trace will contain multiple lines of output to describe the determination of a single data object.  You also know that locating the cause of parsing errors is more art than science. 

 

Well, the good news is that with the introduction of the binary trace viewer, determining errors by viewing the trace just got easier with ITXv9.0.0.3.  I won't bore you with the details, but binary trace has been part of ITX for a while.  For the last several versions of ITX, it has been used to generate the traditional text trace file that you are used to seeing.  Every step of the parse process is still tracked, but with the binary trace viewer, the relevant information needed to identify parsing issues, is shown at the beginning of the trace file produced.

 

For example, let us look at the trace produced by a map that uses the following simple type tree to describe the data shown that follows the type tree.

The data fails parsing because the EstDlvryDate of the 3^rd record is earlier than the PurchaseDate, which violates the component rule.  Using traditional text trace, the trace file generated is 123 lines long, and the component rule failure is not found until line 94, as seen below.

 

Granted, this type of parsing error is easy to find for those of us who have been looking at trace files for years, but it could take a while for a new user to figure out what was wrong, and even experienced users will still spend time to isolate the error.  Suppose the file had thousands of records, imagine how long that may take.

Now we will look at parsing the same file using binary trace, and using the binary trace viewer to look at the output.  While in this example, the total file size is larger than the original trace, the offending data is found at the very beginning of the trace file, as seen below.

 

 

 

 

The entire content of the first 10 lines follows:

Trace Summary:

INPUT 1:

Level 0: Offset 0, comp 1 of 1, #1: Input 1 was valid but contained 56 bytes of unknown data at the end.

 

Last known item found ending at offset 144:

Level 0: Offset 0, comp 1 of 1, #1: Input 1 was valid but contained 56 bytes of unknown data at the end.

Level 1: Offset 105, comp 1 of 1, #3: Component number 1 of type 'File Groups Root': occurrence 3 is optional and does not exist.

Level 2: Offset 136, len 8, comp 5 of 8, #1: Data at offset 136 ('20181010') failed component rule test for component number 5, type 'Record Groups Root'.

This shows that by using the binary trace, and the binary trace viewer, it is much easier to find where the parsing error occurs.

 

As an added benefit, the binary trace is very efficient.  Unlike the text trace, which adds much overhead to map processing, I have tested maps parsing a 40 MB file, and the performance hit was negligible.  The file I tested took 33 seconds to parse with trace turned off, and the same file took 33 seconds to parse with binary trace turned on.  Of course, depending on the complexity of the type tree, your results may vary, but running a map with binary trace turned on is much more efficient than running the same map with text trace turned on.

 

Now that we have discussed binary trace, how do we generate the trace and view the results?  From the Design Studio, it is as simple as turning on trace, setting the option to binary, and then using the organizer to view the result, as shown in the pictures below.

 

To view the binary trace outside of Design Studio, you can readable output from the binary trace from a command prompt by invoking dtxmaptrace.bat located in ITX installation directory.  The screenshot below shows

1. Defining a path to the ITX Installation directory
2. Invoking the dtxmaptrace.bat without input to display usage information
3. Invoking with reference to TestMap.mtr, which was generated by turning on trace, and setting the binary option as shown above.

Now that you can find parsing issues more quickly, what will you do with all your extra time.  One request from our development team is for you to play around with binary trace and the viewer, and then provide feedback to help us to make it better.  As you already know, ITX is able to model data in so many ways, that it is impossible to test every possible data modeling scenario in our labs.  If during your testing, you produce results using the binary trace that you feel produces too much output, please let us know about it.  We want to improve the binary trace capability to make it as efficient and as easy to use as possible.

 

Until next time, Happy Mapping.

 

James Herry

------------------------------
Stephanie Fetzer
------------------------------

#IBMSterlingTransformationExtender
#DataExchange