I am currently working on a project to mask data in CSV format files. This is normally fairly straight forward using Optim Connect and DB2 Fed Server. However, we have come across some complications. One of the fields is a free form text field that can contain any type of character, including commas. Non-numerics are typically surrounded by quotations and quotations are represented in a string as two in a row.
The problem I have seen is that there are some strange values in the free form text fields. For example a field value might look like "This is the ""quoted"" string, which is all part of a single field." Optim connect seems to break on this. To make things even more complicated, I have seen some of the entries look like "This is a ""quoted,"", string, which is all part of a single field". I have no idea why anyone would punctuate like this, but they do and it is our job to work around it. There are also line feeds in the middle of some text statements, which throws it off as well.
The solution we have been working with is to "sed" the special characters into tags. For example, a line feed would be turned into <LF> and the pair of quote marks would be <DQUOTE> and then later putting it back. However, every time we work past one of the problems, we find some other creative user entries that break the sed scripts.
I am considering to write something to parse and tag the special fields. Before I do, I would like to make sure there isn't an easier way that someone else may have done.
Thoughts?
------------------------------
Tammy Forlenza
------------------------------
#InfoSphereOptim#Optim