Originally posted by: SystemAdmin

The only thing I can see that IBM WebSphere TX will do, is map the entire record for one bad field (we do that in 100's of maps). I want to run a second map against the rejected record file to output a report to customer support that gives them the actual field that rejected and why it rejected so no trace is required.

Anyone have any ideas? I haven't been able to make this work. No matter what I try, it will not map the bad field to an output card. I know I can use the audit log, but that means we'll have to permanently turn on audit logs for ALL maps all the time.

#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

Have you used the Reject function?

Only 'valid' objects (even ones with errors) can be mapped using 'normal' functions. Output data is not created for invalid data. The REJECT() function will help as can the iserror and containserror functions.

You must also set the restart attribute in the type tree. The type designer doc can help describe this.

Finally, be careful that this is what you wish to do. I have a client that uses this extensively on DB input; the only time this should happen is when the type tree and mdq don't match, not when DB errors occur.
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

Hello Steve! The Reject function returns the whole record if an invalid field is found.

Have done a lot of extensive validation maps and reporting here and use component rules in the type tree with the Onerror function to write a custom error text to the audit log. Maps can also be used to check for invalid fields but the Onerror function is not available, you have to write your specific rules and send the output to a reject report file.

Not sure why it is an issue to turn on the audit logs for your maps as they do not take up a lot of space, especially if you have only the summary option on for a map.

However, for your particular instance, a possible solution is to create a file of the records that rejected in the first map, run that output in your second map which will have a tree defined with component rules and then you can validate individual fields and write error text to the audit log, turned on in the second map, with the Onerror function.

Possibly another solution is to have a second map read in your file of reject records from the first then put validation rules in the output card and write your error text to some type of reject report file and this would not need the audit log.

Key point if using the audit log and the Onerror function is to use the Data Audit Settings tab in the Organizer to track a record level so that the audit log will give you all the information necessary.

Have used a Perl script process here to parse out the E09 errors from the log and create a nice error report but have finally figured out the audit type tree and will be using that to parse the E09 errors and create a report directly with a TX map.

More ways to do validation probably exist but in working with Professional Services and using TX to its fullest, even with it not being built for easy validation reporting, it gets the job done now.

The Professional Services consultants that have come here to help with other issues have indicated that they have not found any other organizations using the extensive validations coded here, but the available validation features and functions in TX have been used to their fullest and the applications work fine.

Now Ascential also has a Profile Stage tool to profile your data which is being looked at here too and that is supposed to make it easier to validate data up front.

For now the TX tool is doing the work and the way you choose to validate is personal preference given the current functions available in the tool.

Hope this helps.

Thanks!
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: SystemAdmin

Thanks for the feedback. Gives me a good bit to go on. Turning the audit logs on isn't a systems resource issue, but rather a analyst resource issue. Anytime we touch a map we're required to retro-test and go through a pretty rigid implementation process. To change all of our maps can take up to six months.

If TX would had a function to map an invalid field (if you wanted to) it would be easy. We use the REJECT function extensively now to kick out these bad records but when the "report" goes to our support dept., they can't interpret what the error was just by looking at raw data. I want to create reports that say "this record failed because <x> field was invalid".

Thanks again.
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

I recommend turning summary audit on always and routing the audit logs to a directory where another map reads the audit logs and if there are errors moves it to an error logs directory and sends an alert if applicable. The summary audit very little performance impact, if any.

I don't recommend turning on data level audits across the board as this does have a performance impact. Suggest using the reject function to capture the bad data and then have a validation map that can be run in a test environment on demand with data audit and/or trace on that can then be used by customer support to identify the problem.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

If you use audit log maps, I would recommend only writing them on error. There is a switch in the audit log settings. I think this became available in 6.7.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

Good point re: Audit on Error. If you are not going to use the successful audit logs for anything (stats, process confirmation, etc.) there is no point in creating them.

Note that Audit on Error and its cousin Trace on Error have the same performance impact as the normal settings. (I.e. Summary Audit has little impact, Data Audit can have impact, Trace has impact. The "on error" just determines if the output is created.)
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: SystemAdmin

I set up the audit log to write on error and it does. Problem now is I have to be able to associate that back to a specific record by some field, like loan number or invoice numer. So the support staff will know which record it's referring to.

I'm trying to avoid using the trace because that means support would have to have a Design Studio license, access to all source files, and the knowledge to turn on traces and read them. Too involved for first level support.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: SystemAdmin

Hello Steve! The Onerror function has a parameter that allows you to send text to the audit log.

For example, the following makes sure the last name is not blank then sends the SSN, key field, plus an application error code to the audit log.

ONERROR (TRIMRIGHT(DEP_LAST_NAME Field) != "",
LEFT(EMPLOYEE_SSN Field + " ",9) + " GD080")

The E09 statement in the log for this field would display blanks for the data in this field, followed by SSN and error code

123456789 GD080

This is assuming you are using the Onerror function in a component rule in the tree. To have the SSN or any key field included in your error text the field must already have been read in the tree, it must be prior to the rule where you want to display some error text.

The tree analyzer will give you an error if you try to use a field in a rule that is a later component number in the tree. The Onerror function allows you to use 32 positions for error text, any more and the text in the log will be truncated.

Have asked Ascential a couple of times to increase this length in a future release since 32 positions is not a lot and therefore abbreviations and the like are needed.

However, the application error code has helped in that this is matched to an error code description table to display a more detailed error text when the error report is created.

Hope this helps.

Thanks!
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: chinna508

PLEASE PROVIDE ATTACHMENT FOR REJECT & ON ERROR FUNCTIONS

#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange