Data Integration

I am working with the Hierarchical stage in Datastage flow designer to access SharePoint lists via the RestAPI. I have written some powershell to generate a json schema file from the fields output of the list. However the Hierarchical stage does not like this resource file. All documentation I can find on this states to use the Schema Library Manager, which is flash based and therefore not available to me. I feel like I should be able to manage this through the Flow Designer, rather than the IBM® InfoSphere® DataStage® and QualityStage® Designer, I just need to know what json file should look like. Below is the error that I get when importing my json file.

Attachment(s)

SP_TestList_Schema.json   34 KB 1 version

Hello again,

You might be seeing the issue that was fixed by this APAR:
JR64973: Hierarchical Stage: REST step throws invalid xml error when xml response body contains less than 4 characters in the first line.
The above APAR is fixed in IIS v11.7.1.4.

Here is a paste from the README file in that patch:

PROBLEM DETAILS:
===============
In Hierarchical Data stage, REST Step throws invalid XML error when the received XML content contains 
less than 4 characters in the first line of the body content. This happens because REST Step uses 
the first four characters before a new line for determining the charset.

Following is the error message
CDIER0925E: The REST step found invalid XML data in an HTTP body.

RESOLUTION:
==========
A new java property xmlcc.ignore.invalidXMLError has been added, setting this to true will by pass the
check and assume the charset to be UTF-8 and proceed. Please note that this does not guarantee that 
the resulting XML will be valid and may not be parsed in the XML parser step.

Regards.

Original Message:
Sent: Fri October 20, 2023 02:52 PM
From: Emily White
Subject: Hierarchical stage - Flow Designer - json libraries resource file structure?

Hello,
Which version of Information Server are you using?

 

There were a couple of known issues with DFD returning the generic "RestApiResponseUnsuccessful 200" error in some situations, because it is missing the default directory or the default project. Please see the links below. The issue that you raised might be related,  and the issues have been fixed.

 

JR61096: THE DEFAULT DFD DIRECTORY ._DFDESIGNER IN ENGINE TIER WAS NOT CREATED DURING IIS INSTALLATION.

https://www.ibm.com/support/pages/apar/JR61096

 

JR61281: DFD API IS LOOKING INTO DEFAULT "DSTAGE1" PROJECT WHEN NO PROJECT IS PASSED TO THE API.

https://www.ibm.com/support/pages/apar/JR61281

 

That said, according to this link 

https://www.ibm.com/support/pages/node/1282666 

the Datastage flow designer (DFD) approach for the Hierarchical Data stage is for simple use cases.

Simple use cases can be

• Parsing and Composing XMLs which can be just imported/referred (cannot apply schema chunk operations).

Working scenarios:

• JSON Parser and Composer

• XML Composer

• REST 

• The following transformation steps work fine:

  OrderJoin (change in usability) 

Non-Working scenarios:

• XML Parser

• The transformation steps: Regroup, Switch.

The error message in your screen shot says there is an Unexpected character. Can you check your json file with a hex editor for any kind of garbage or unprintable characters?

Regards.

------------------------------
Emily White

Original Message:
Sent: Thu October 19, 2023 11:03 AM
From: Jackson Eyton
Subject: Hierarchical stage - Flow Designer - json libraries resource file structure?

I am working with the Hierarchical stage in Datastage flow designer to access SharePoint lists via the RestAPI. I have written some powershell to generate a json schema file from the fields output of the list. However the Hierarchical stage does not like this resource file. All documentation I can find on this states to use the Schema Library Manager, which is flash based and therefore not available to me. I feel like I should be able to manage this through the Flow Designer, rather than the IBM® InfoSphere® DataStage® and QualityStage® Designer, I just need to know what json file should look like. Below is the error that I get when importing my json file.

I have also attached my attempt at the json schema file. Any help would be greatly appreciated.

------------------------------
Jackson Eyton
------------------------------