Optim - Test Data Management and Archiving

Processing large clob column value in TDM column map exit routine

  • 1.  Processing large clob column value in TDM column map exit routine

    Posted Tue March 09, 2021 03:56 PM
    I'm working on a new TDM column map exit routine written in C++/C for an installation of distributed Optim TDM, based on a codebase that was written for us several years ago by an IBM consultant.  I have only a rough passing knowledge of C++ so I am kind of hacking my way through this code.  Managed .NET C# is much more my forte.

    The exit routine will process an XML document stored as a clob within a SQL Server varchar(MAX) column, using ODPP to mask specific nodes within the XML doc. I actually have the exit working as intended, with one problem.

    My DLL crashes when trying to process rows where the clob size is > 400,000 bytes. I have a hard-coded limit so that my exit immediately bypasses rows where this clob size is too large; this limit-check was carried over from the older codebase I mentioned earlier. When I remove the limit check or increase the limit above 400,000 my DLL crashes; when I keep the limit at 400,000 it runs successfully but I reject/bypass about half the rows in the source table. It is a requirement for this particular application that I keep and process all source rows.

    We are running distributed TDM v11.3 on a Windows server. My exit code is using the various standard ODPP libraries, header files, etc. from our Optim installation. The farthest I can trace the problem to is struct s_PST_CMExitParm function pPSTGetLOBSegment, defined in PSTCMXIT.H. I'm passing in 400000 as the 3rd input parameter for the 'Length of the LOB data segment to be returned'.

    Other than the pPSTGetLOBSegment input parameter for LOB length, I'm not seeing anything that I can change to increase the allowable size of the clob; no switch, config setting, etc.

    Any guidance, tips, suggestions, etc would be much appreciated.  Thanks.


    Snippet of s_PST_CMExitParm from PSTCMXIT.H:
      /* The following function may be used to get substrings from a LOB
         source column. */
      short(*pPSTGetLOBSegment)(short,    /* Col Handle for the Source
                                              LOB Col */
          long,     /* Zero based offset of the
                       1st byte of the LOB data
                       to be returned. */
          long,     /* Length of the LOB data
                       segment to be returned */
          void *,   /* Pointer to the output buffer
                       area. It is assumed this
                       buffer area is at least as
                       long as the requested segment
                       length. */
          long *,   /* Pointer to a return value
                       that will indicate how
                       long the full LOB data is */
          long *,   /* Pointer to a return value
                       that will indicate how
                       many bytes were actually
                       returned. This may be shorter
                       than the requested length
                       if the requested substring
                       exceeds the end LOB data or if
                       a Multi-byte char of NCHAR
                       character would not be fully
                       returned. */
          short *); /* Pointer to a short Indicator
                       variable. After the call, this
                       field will contain -1 if the
                       column is NULL. Otherwise,
                       this field will be zero.
                       This parameter may be NULL */

    ------------------------------
    Rob Searson
    Application Developer
    Progressive Insurance
    Mayfield Village OH
    ------------------------------