Hi,
I am trying to grab Authorization key which is from a document field and then its value

Document field looks something like this:

{

"document": {

"protocol": "http",

"subprotocol": "HTTP",

"http": {

"requestUrl": "/rest/test",

"method": "GET",

"requestHdrs": {

"Accept": "application/json",

"**Authorization**": "**Basic whatever**"

},

"ipInfo": {

"localPort": "5555",   

"remotePort": "64880"

}

}

}

}

image918×237 10.9 KB

image983×117 5.05 KB

Basically loop through entire document then within the document loop create another loop for http then within http loop create another loop for requestHdrs to eventually grab Authorization key and its value.

Thanks

you can write nested loop. drag loop inside loop. i won’t suggest doing that though… loops are costly operation.

btw from the picture, i don’t see any document as loop so it may be just matter of straight map

Actually looking for code snippet:

Here is an example
Document field in webMethods looks something like this

In webMethods there is a different way to loop through this field

Something like this
This is the loop to read through entire document field

IData AllConvertedHeadersToDocument = IDataUtil.getIData( pipelineCursor, “AllConvertedHeadersToDocument” );
if ( AllConvertedHeadersToDocument != null)
{
IDataCursor documentCursor = AllConvertedHeadersToDocument.getCursor();
while(documentCursor.next()) {
documentCursor.getKey()
}

}

Now I am trying to read below

https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/373c7887296ed0f50b5da06f09c1ec208ec3433f.png 1.5x, https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/373c7887296ed0f50b5da06f09c1ec208ec3433f.png 2x" data-dominant-color="F2F2F7">

image852×169 34.4 KB

{
“document”: {
“protocol”: “http”,
“subprotocol”: “HTTP”,
“http”: {
“requestUrl”: “/rest/test”,
“method”: “GET”,
“requestHdrs”: {

            "Accept": "application/json",
"Authorization": "Basic whatever",
"Cache-Control": "no-cache",
"Postman-Token": "0806e356-767c-482d-a1f9-9c7c5636559d",
"Host": "localhost:5555"
}
}
}

}

Trying to get requestHdrs level.

This is an in built feature.
Map is not going to work with in built feature.

Thanks

Hi Mohammad,

I agree with Mangat, that there is no looping needed in this structure.

The fields marked with a Document icon are just for grouping fields.
So you can try to access the source field as “/document/http/requestHdrs/Authorization” and just map it to the target.
As an alternative you might want to expore the WmPublic package for the folder “pub.xml”. In this folder you will find a service for querying documents
See IS Built-In-Services Reference for details.

Regards,
Holger

Hi,

Folks I am having a rough time figuring this thing out.
Here is the run down

I am using
to get all headers. The result comes out in a Document field.
I create a java service and create a Document field and mapped the passed headers from above in built service into the newly created Document field for my java service for further processing.

I right click my java service and generate code with all fields
It generates the code something like this
// pipeline
IDataCursor pipelineCursor = pipeline.getCursor();

		// GrabbedHeaders
IData	GrabbedHeaders = IDataUtil.getIData( pipelineCursor, "GrabbedHeaders" );
if ( GrabbedHeaders != null)
{	
}
pipelineCursor.destroy();

IDataUtil.put( pipelineCursor, "Token", val );
pipelineCursor.destroy();

Document field stores result as key and value pairs

image903×212 9.8 KB

What I want is to grab Authorization key and its value.

This is the Document Field where all the passed Headers are stored
IData GrabbedHeaders = IDataUtil.getIData( pipelineCursor, “GrabbedHeaders” );
Its called IData
I dont see any method to grab the key and its value in below screen directly:

I hope you folks understand where I am getting at.
Its very simple I need to grab the Authorization key and its value.
Please provide if you have code snippet.

Thanks

Mohammad, for this requirement why do you even want to create java service? is there a reason that you absolutely need to write java service.

I need to decode the token and perform validation. I am a Java developer but newbie with webMethods.

Thx

okay… you don’t need java service for this.
look into map. this is a straight map. input to output pipeline

Thanks everyone for the response but nothing worked.
Came up with a solution with so much playing around with the field itself.

Here is the solution:

    // pipeline
IDataCursor pipelineCursor = pipeline.getCursor();

// Headers
IData	Headers = IDataUtil.getIData( pipelineCursor, "Headers" );
if ( Headers != null)
{
IDataCursor HeadersCursor = Headers.getCursor();

// i.http
IData	http = IDataUtil.getIData( HeadersCursor, "http" );
if ( http != null)
{
IDataCursor httpCursor = http.getCursor();

// i.requestHdrs
IData	requestHdrs = IDataUtil.getIData( httpCursor, "requestHdrs" );
if ( requestHdrs != null)
{
IDataCursor requestHdrsCursor = requestHdrs.getCursor();
String Authorization = IDataUtil.getString( requestHdrsCursor, "Authorization" );						
requestHdrsCursor.destroy();
}
httpCursor.destroy();
}
HeadersCursor.destroy();
}
pipelineCursor.destroy();

// pipeline
IDataCursor pipelineCursor_1 = pipeline.getCursor();
IDataUtil.put( pipelineCursor_1, "Result", "Result");
pipelineCursor_1.destroy();

Thanks

For those coming upon this thread later, do not follow the Java code shown. Using Java is not necessary for this.

There are a couple of techniques to use in FLOW to easily get the value of any header. Here is a simple example.

Call getTransportInfo as the OP ntoed.

image682×540 23.2 KB

The trick is to manually add the named var you want to the untyped document in the pipeline. Right-click on the doc and insert a string var named Authorization.

image744×781 25.6 KB

Then create another var in the pipeline and map from the source field to the target field.

Now you can do whatever you want with the value in authorizationToken. And requestHdrs can be dropped.

HTH.

Sufran – you need to resist almost 100% of the temptations to drop to Java. Most everything can be done in FLOW. There are certainly times when Java is needed/useful but you should endeavor to have that be a rare occurrence. This old, old post may have useful guidance/info. Integration Server and Java - #22 by reamon

Yes Reamon, when I came up with this solution I was newbie at that time, actually still am. I started working on webMethods a month back or so. But you are right. I found out the solution later that you have mention above using direct mapping but at that time I didnt know.

I want to piggyback on what you just mention about limiting the java code and use flow more.
And this is regarding Json schema validation that you provided a way out for me.
I am on the next level now. Hence the question is, when provided schema that I am validating throws some errors on fields for format/required check. They are default errors comes inside list as bunches. Is there anyway I can convert them into custom errors without using any java code at all.
Here is the example of default errors are thrown inside list:

image1129×301 10 KB

So for example
I want to convert this
[ISC.0164.0030] JSON schema validation failed: required key [logo] not found
to something like
Logo cant be null or empty

another one
[ISC.0164.0030] JSON schema validation failed: string [12345678] does not match pattern ^[1]{9}$

ID must be 10 digits.

I want to convert all of them errors into custom errors without using any java code at all in this case.
Any clue Reamon?

Thanks

1. \d ↩︎

Glad you had determined how to do the mapping in FLOW earlier. It does take some time to learn how to use FLOW.

For the errors item, you can create a service that does the translation. Loop over the errors document list and for each do whatever fits for how you want to translate. A couple options:

• BRANCH on errors/error to create your own message. However, it looks like the error codes may not be granular enough for what you want. E.g. JSV17 is likely the same code used for any missing field, not just logo.

• BRANCH on errors/message and use regex in the label to determine the specific error that occurred. Then you can return any value you’d like to describe the error. You could possibly parse the message value using pub.string:tokenize to split the string on the : character so you could omit the [ISC… part of the message. Be aware of what tokenize will do if : appears in the error message itelf – particularly if payload data will be present. May need to “reconstruct” the string. You could also use regex with pub.string:replace to strip the message prefix.

• Define a lookup table. Use pub.string:lookupTable, possibly with regex, to do the translation.

• One thing you may find helpful, and gets some Java coding for you is to create a couple of utility services. Most companies create their own “public” package, a la WmPublic, to define various common utilities. A couple we have are splitString, which simply exposes String.split(), and matcher, which wraps java.util.regex.Pattern and Matcher. Something similar may be helpful for what you’re doing here.

I’m obviously not familiar with what your top level service is intending to do, but I would strongly recommend against doing too much in terms of validation and customizing the error messages. You’ll likely spend an inordinate amount of time on error messaging rather than on functionality.

Is your top-level service providing the functionality directly? Or is it passing the request data on to another server/component that does the work? If the latter I would strongly recommend against doing validation “in the middle.” Let the target component worry about what it does and does not need.

Reamon thats a lot of information to digest.
However to answer your question, As a provider when Json request body comes in, it has to be validated and anything fails, you are not getting any further and all the validation errors will be thrown back until every input in the request body is fixed. You are stopped there. Thats the whole idea.

Is wM IS really the complete “provider” or is it more a front-end providing access/minor transformation to something else? My query is geared toward understanding what is done with the JSON payload. Will it be passed on to another system, then it responds and wM IS passes the response on to the caller? Is all the processing being done on wM IS? Or is it doing some initial work and calling another system/app?

If the wM IS components are passing the data on to another app, via their API, what will that app do in the face of missing/incorrect values? If that app returns an error, you might consider just letting it do the validation.

In other words, if you’re creating components in wM IS to “protect” calls to another application you may not want to do that. Let the application protect itself. It knows its rules. No need to repeat them in wM IS.

But as noted, I’m not familiar with the solution you’re building so this suggestion may be off base. My intent is to maybe help you avoid current/future pain. This is definitely in the YMMV category, but in the past I’ve done this very thing, validating every incoming document before handing it off (some transformation perhaps) to another system. Just about every time we’ve done that in wM IS, it came back to bite us and we removed it and had the end point handle it. Of course there are times when that isn’t possible – end point is a DB table. But if it is an app API, let them enforce the rules.

As a provider, our first concern is to protect our API. We dont want any garbage or malicious information. We will stop you there. So plan is to do protect/validate and massage data to pass it to backend service for further processing and then whatever we receive we have to do conversion to pass info to the service that is calling us. So its like chaining process. It will be like a microservice. The difference is we are not using any frame work like Spring boot.

So there is alot to be done here. My next goal is to handle exceptions.
I tried your suggestion.

image794×126 7.97 KB

Result

image747×140 5.58 KB

I got close with this. But like you said might have to build some utility services to accomplish some intricating work.
Glad you are here to usher me. Appreciate your help very much.
I am going to be putting alot more postings in upcoming time.
I am new at this. So I am learning and implementing as I go.
Its a process but candid truth I really like this WS designer. I want to build on this and get better. Snippets/codes/screen shots etc will help alot more for better understanding. you probably seen in every one of my post, I provide screen shots/codes that I am having issues with. So that person that is trying to understand your problem will have a better understanding. I understand most of the stuff that I am doing is common/advance work but it is the way in real world.

Thanks

Sure. A provider should protect itself. But my question is asking whether or not the wM IS-hosted component is “the provider.” Based upon what you’ve shared, it seems likely that it is not. It seems that it is providing another “interface” to an existing provider. What is the nature of the “backend service?” Are you calling a defined API? What does it do when given bad data? I ask because if it is already “protecting itself” adding “protection” in wM IS is not necessary and simply adds complexity for no real value.

If the “backend service” being used already validates data and returns error messages, leverage that. Don’t redo it. Keep the wM IS interface component as thin and light as possible. Be “pass-through” as much as possible for request/reply interactions. Don’t add redundant capability.

Other random notes:

• Don’t overuse transformers in a MAP step. Specifically, don’t use a MAP step that has just one transformer call in it. Just INVOKE the service. Helps with readability.
• Don’t be misled by the wM documentation that infers that transformers run in their own thread. They don’t. The caveat they give is intended to indicate that the order of execution of transformers in a single MAP step is undefined.
• Use caution with a hard-coded index. Any change at all to the message format, including validate adding just one more digit to the error code, will break it.

I am not sure of that at this time on where it will hosted eventually once it goes to production. We are going to call another Java service that is actually calling Oracle DB. They are established JavaEE services. They throw there own custom error codes and we take them and convert them to our own custom code and pass it on. Basically I dont have control over that. Thats just the requirements.

That’s what we are trying to do, keep everything thin as possible but we can only decide as we move further down the line, what to keep and to remove.

Are there other alternatives to MAP that I can leverage Reamon? what if I want to do some conversion during looping process or add some checks using branching, Is there any other way to attain that?

they throw there own custom error codes and we take them and convert them to our own custom code

What is the value of converting the error codes? Understanding that “requirements” is an overloaded term, we often forget that some “requirements” are really just design decisions. If it were me, I would press for why error codes/messages would need to be changed.

As example, does changing [ISC.0164.0030] JSON schema validation failed: string [12345678] does not match pattern ^[1]{9}$ to string [12345678] does not match pattern ^[2]{9}$ really matter?

In any case, there are many ways possible to translate. Think about how you would do this in Java (but don’t do it in Java) and translate that to doing the same in FLOW. BRANCH statements (same as switch/case). Lookup table. You could externalize the lookup table by defining it in an XML or JSON file and loading it at service execution. Or in a DB table. You could use regex where appropriate for string matching. Create focused helper functions as needed for matching/replacing. It all depends on how sophisticated and complete you need to be with translating the errors returned by the Java EE components.

1. \d ↩︎

2. \d ↩︎

Hey Reamon,
Sorry for the delayed response. Stuck on new issue now. trying to work through that.

To answer your question hence basically it is a standard, what code is coming from backend it has to be converted into code that is understood by users. Basically they are already defined, when we do conversion, we just set the closest one that almost or nearly matches the backend code or if we don’t have one, we define one. Lets say something like, the most famous one null pointer exception, if we get that from backend, we want to convert that into something that user can understand. Thats just an example or something like bad request.

Also I found out it will be running on-premise server.
I will look into lookup-table on how to best use of that.

Reamon I am working on removing all the hard code values and going for the approach to use properties file like application.properties.
Do you know what I can use/in-built service to call properties and grab values using keys.
Any idea what should be the approach.
I am thinking I will create a property file, I dont know WM supports what type of properties yet. I am doing some research now.
Perhaps if you have any idea, please let me know.
Also I approach will be to grab values based on Environment, like get local value if I am running locally, dev value if I am running in dev, production value if running in production and so on.

Appreciate you for all your buttress. Thx

I found two ways:

first method:(perfect solution for env based values)

on IS server
grab values uisng

second method:(perfect solution for just local)

this help you grab it from your C drive

Hi Sufran,

additionally you might want to check out for the Global Variables feature available since late wM 9.x versions.

Regards,
Holger

Hey Holger,
Thanks Holger Global variables eliminated several lines of in built service uses. Worked flawlessly. Thx much

Just in case,
Anyone who wants to know how to create one
Go on IS server and under settings and click


add one entry
Then you can call it
Double click on any field you want to set the value to

check this box
this tells you have some global variable set to this

Thanks for the additional info. I’m still concerned that converting the errors is more than needed. When you say “users” to you mean people using an application/browser UI? If so, are your wM IS-hosted components formally a part of that application? Usually wM IS-host components are not. They are “backend” type things. The application itself should translate as needed to get “user-friendly” error messages, not the wM IS items. The wM IS audience is usually developers – API client. But I think we’ve probably beat this specific item to death.

For configuration settings:

• Use the the extended settings in IS extremely sparingly. That is not really for our use. In our environment, we have only a couple of custom items in extended settings. And only because there was no “Global Variables” facility at the time we started. Use Global Variables, not extended settings.
• Holger noted the Global Variables feature. A good facility for the right types of config. Don’t go crazy with this either since it is global. Easy to get to a point where people are afraid to change anything as they don’t know what it might break. Need naming and usage discipline.
• External files are the primary way to go. Here is a common convention:

1. Define the files in a common location. We use /instances//properties.
2. Do NOT place the files in the package directory. It is very tempting to place the config file in the package config or pub directory but you’ll likely regret doing so later. Make the deployment of packages and config files separate. That way, an updated package deployment won’t clobber any config file changes that may have been made in a specific environment.
3. The config file content can be whatever you’re most comfortable with. Classic properties file, loadable with java.util.Properties. An XML file, which is what we use. A JSON file. Or possibly an .ini file, which is conceptually a properties file with the added notion of sections (probably not a strong choice). Use a file naming scheme that makes sense for how things are structured. Package based perhaps, depending upon the scope of your packages.
4. Define a document type that corresponds to the structure/elements in the file.
5. Define a common service that can be used for any arbitrary config file (for a given format). Various packages can call this to get their config. The service returns a document, with a structure matching the document type defined.

Here is a simplified set of steps for loading an XML file:

Basically the same steps when processing any XML. Just loading it from a local file. Be sure to pass in a document type name if the config file will have repeating elements.

For JSON, quite similar but use jsonStringToDocument instead of xmlStringToXMLNode/xmlNodeToDocument.

For properties, create a Java service to read the properties and convert each property to a field in an IS document. Here is an example which supports a caller passing a stream, bytes, string or filename and returns an un-typed document which the caller can map to a document var that has a document type reference. The key for that is the doc type definition and the file content need to match.

public static final void propertiesToDocument(IData pipeline) throws ServiceException {
IDataCursor idc = pipeline.getCursor();
java.io.InputStream stream = (java.io.InputStream)IDataUtil.get(idc, "stream");
byte[] bytes = (byte[])IDataUtil.get(idc, "bytes");
String s = IDataUtil.getString(idc, "string");
String filename = IDataUtil.getString(idc, "filename");
boolean closeStream = false;
idc.destroy();		

// Figure out which input was passed.
if(stream == null)
{
if(bytes == null)
{
if(s == null)
{
if(filename != null && filename.length() > 0)
{
try {
stream = new java.io.BufferedInputStream(new java.io.FileInputStream(filename));
closeStream = true;
} catch (java.io.FileNotFoundException fnf) {
throw new ServiceException(fnf);
}
} else {
return; // Nothing to convert; return nothing
}
} else {
stream = new java.io.ByteArrayInputStream(s.getBytes());
}
} else {
stream = new java.io.ByteArrayInputStream(bytes);
}
}

// At this point we have a stream (bytes, string converted to a stream)
java.util.Properties props = new java.util.Properties();

try {
props.load(stream);
} catch (java.io.IOException ioe) {
throw new ServiceException(ioe);      
} finally {
if(closeStream)
{
try {
stream.close();
} catch (java.io.IOException ignored) { }
}
}

IData document = IDataFactory.create();
final IDataCursor dc = document.getCursor();
props.forEach((key, value) -> {
IDataUtil.put(dc, (String)key, (String)value);
});
dc.destroy();

idc = pipeline.getCursor();
IDataUtil.put(idc, "document", document);
idc.destroy();	
}

HTH!

Hi Rob,

thanks for your update.

Addendum:
We are using a mix of classic properties files and XML config files, which are located in dedicated subdirectories of IntegrationServer/instances//config folder.
This has the benefit, that these will be backed up during successful IS restart by the internal logic to config/backup subdirectory.

Access to global variables should be able by built-in Services documented in the Built-In-Services Reference similar to pub.utils:getServerProperty.

Regards,
Holger

Hey Reamon,
First thing first, conversions of default error is imperative to provide user friendly code mainly for understanding purpose. As you already can tell these default error code in webMethods are not user friendly at all. From developers perspective not a biggy but from users point of view a little perhaps. So no way out of this but to add some customization to these default errors thrown when failure happens at Schema validation level. Now imagine as I already mention I began to work on the exception thrown part when and if there is any 4xx to 5xx thrown. There are default exception thrown. So I have to customize them as well. Any thoughts on that?
Ton of work to be done here. No time to sleep

Then regarding the UI, I think I misled you a little on that. This orchestration will be build as API/Service rather and will be part of another layer that will be called.
The idea is it will be like a rest service that is internally can call either DB or Web service(Soap/REST). Sky is the limit in this case.

Everything is tentative at this time but approach is just like creating a Spring boot app.

Now going towards using properties like Spring boot we have externalized properties.
So I went with this approach

And then came Global variables and you can see how many lines I have disabled by just using Global variables. You can say I am in love with Global variables now.
But idea behind this is to initialize values at start up, convention or pick up some values down the line when doing further processing in code. FX: DB user/password/driver etc. Global are perfect for these situations. But I do want to use external files in parallel. However there is slight problem I am coming across with this approach.
So check this, I go on IS server

set a key value pair

Key is something like this
watt.custom.path=C:\Home\users\test.txt
Now what I do is use this

in built service. It takes key and return value

Then I use this to call the file

and you can image the rest.
Now the problem is I will be passing the key as parameter because it will called through rest service endpoint, So I need to use this first

Now this gives me parameter directly which is basically my key(watt.custom.path) and I map it to this

and I get key back but now the problem is here
I get the path back with double back slashes
C:\\Home\\users\\test.txt
It wasnt happening when I wasn’t using this

Its weird.
I tried using this in built service

no effect.
This defeats the endeavor here for external files calling.
So I will try second approach you have provided with code
create a Java service to read the properties.
Lets see but I want to use the external file mechanism as well to be more limber. Like you said dont go heavy on Global variables.
I am glad you here mate to help me out.

Much RESPECT.

Hi,

it turned out as a best practice to specify all pathes in wM with simple slashes (‘/’) regardless of OS.

Global Variables are not always the best choice.
They apply only to those keys which are environment specific, but static during runtime.

For more dynamic keys properties files and/or XML-based config files might be the better choice.
If you do not want to read these files from file system every time they are accessed, you can choose the built in caching functionaliy. But make sure to set the cache expiration to a reasonable time so they are refreshed regularly.

Regards,
Holger

Thanks Holger. Forward slashes worked even though I have windows 10 OS running. Its weird with when its Linux way.
I will try cache functionality as well. Thx

Be careful with this. The caching facility captures the entire pipeline at the time. So if you’re 10-20 steps deep into a service and call a service that has caching, it will capture everything. Then the next run, that cache will overwrite everything in the pipeline – likely causing undesired behavior.

Use caching sparingly (don’t optimize prematurely) and when it is used, use the scope feature to limit what the cache contains – so that it caches only the inputs/outputs of the service being cached.

Hi,

or invoke the service being cached as a transformer as this will only capture the exact input signature of the transformer invocation and not the full outer pipeline.

Regards,
Holger

It depends on the “user” audience. For APIs, the audience is developers, not app users. Messages for app users is the domain of the app, not the API.

For the 4xx, 5xx HTTP status codes, what you do depends upon what the server being called returns. Even for 2xx status you may need to do something more than just check the code. The HTTP body may contain additional information, of course depending upon what the server does in various scenarios. Some will return 200 but indicate an error in the HTTP body.

Those lines should be in a separate service. A service in your “public” package that can be used by any number of integration-specific packages. With that, you’d then have a one-liner in your “do-the-work” services to load configuration for that service. Don’t overuse global vars.

DB user/pw/driver will be configured in the JDBC adapter connection pool, not in any custom config file or global vars. You’re using the JDBC adapter I assume? Hopefully not Java services.

You’ve mentioned pub.flow:getTransportInfo a couple of times. May I ask what you’re using that for? If it is to retrieve the URL parameters, it is not necessary to use that service. The URL parameters are already in the pipeline when the service is invoked via HTTP.

Hey Reamon,

So what are you recommending, Shall I just keep the default error messages when doing validations on request schema coming in?

On 4/5xx errors, I havent started working on it yet but almost there to start working on them. Because I do have to provide custom response when there is success/failure. Stay tune on that

For Global variables I decided I will be using it mainly in Environment base like Holger mention. JDBC was just an example. However I found another option. What if I want to read some property file, mainly with properties extension
//
String user = null;
String pass = null;

	// pipeline
IDataCursor pipelineCursor = pipeline.getCursor();
String	FileName = IDataUtil.getString( pipelineCursor, "FileName" );
String	getUser = IDataUtil.getString( pipelineCursor, "getUser" );
String	getPass = IDataUtil.getString( pipelineCursor, "getPass" );

try{
// read in the file stream and convert it to a properties object
FileInputStream configFileInputStream = new FileInputStream(FileName);

// properties object is already defined in the Shared code
Properties properties = new Properties();
properties.load(configFileInputStream);

user = properties.getProperty(getUser);
pass = properties.getProperty(getPass);

}catch ( FileNotFoundException e ){
// throw an error
throw new ServiceException("ERROR FINDING PROPERTY FILE: "+e);
}catch ( IOException e ){
//throw an error
throw new ServiceException("Error reading "+FileName+" property file: "+e);
}catch ( Exception e ){
//throw an error
throw new ServiceException("Error while registering "+FileName+" property file: "+e);
}


pipelineCursor.destroy();

pipelineCursor.destroy();

// pipeline
IDataCursor pipelineCursor_1 = pipeline.getCursor();
IDataUtil.put( pipelineCursor_1, "user", user );
IDataUtil.put( pipelineCursor_1, "pass", pass );
pipelineCursor_1.destroy();

This will help me read properties key/value pair values from .properties file.

Now I am in the process of reading a XML file. Not fairly there yet. Stay tune.
So its either .properties or .xml. There are the two files we mainly read from.

The reason I am using this pub.flow:getTransportInfo is to
get Custom required headers value can be multiple custom header fields
get Authorization field
get JSON request body

These are the three things I retrieve from using this pub.flow:getTransportInfo mainly.

With what you mention about caching, It seems like a problem. I will hold off on that. I think Global variable and the code I mention above to read .properties file will suffice my need for now. Only thing left is XML file which is work in progress.

Thx

I recommend that you don’t do validations and let the “backend” worry about it. Have wM IS pass the data to the JavaEE components and then return whatever they respond with – data response or errors.

But I suspect you feel the JavaEE components need to be “protected” for whatever reason, so the next best thing would be, yes, just keep the validation messages as they are. It is unlikely that mapping them to other values is a meaningful improvement.

Hmm. Is the code I shared earlier insufficient? It will read any string, byte array, stream or file that contain properties (as defined/supported by java.util.Properties) and return a document containing fields that are named exactly as they exist in the properties file. This is a general purpose way of getting config/properties into an IS document. Other services can call it passing whatever they have (usually a filename) and get back an IS doc that has all the settings. The IS doc can be described by an IS document type - the caller of the service would map the output document to a new var that is a doc reference to the doc type (think of this as type casting) so they can easily use the config fields and values. It is not necessary for the caller to pass the field/properties names. The service I shared gets them all.

Here again, the steps shared earlier do this.

Get headers – good.
Get Authorization field - good (though you should research to see if built-in components can interpret/manage this for you so you don’t have to do anything explicit.
Get JSON request - not so good. You cannot get the HTTP body from there. And you don’t need to. It is already in the pipeline. Also, you can get the raw URL string from there (http/requestUrl) but you don’t need to. The HTTP body (if present) and the URL parameters will already be in the pipeline when your service runs. IS content handlers do the work for you.

For the HTTP body, the variable that is present will depend upon the Content-Type. For JSON, application/json, IS will place a var the name of which depends on the setting of watt.server.http.jsonFormat in IS Administrator | Extended Settings.

When Content-Type=application/json the variable in the pipeline is controlled by two things:
* watt.server.http.jsonFormat setting
* the jsonFormat parameter in the URL (typically will not use this though)
- when jsonFormat is:
* parsed: document with a name matching the name in the JSON – e.g. JSON converted into an IS document
* stream: jsonStream
* bytes: jsonBytes

I recommend setting watt.server.http.jsonFormat to stream so that any integration that may need to handle large JSON payloads can potentially do so. If set to other values, it makes it more difficult to support when a service needs the data in a stream (and not all in memory at once). The var is usable in your service right away. Just define it on the Input tab. At runtime, IS will populate it. Then you can do whatever you need with it – convert it to a JSON string, parse to an IS document, pass it on to something else, etc.

While this covers JSON, the same approach can be used for other types like XML and delimited files. IS can be configured to “help” as much as you want to hand the data to your service.

For the URL parameters, IS parses those and places each in the pipeline. All you need to do is declare them on the input tab of your service so that the FLOW steps “know” what to vars to expect and handle them as desired.

Using getTransportInfo has the drawback of making debugging more difficult. When debugging, it will not return the same elements that are returned when the service is invoked via HTTP. There are techniques to deal with that, but we’ll leave that for another time.

Another side trip – security folks are likely to object to storing passwords in any text file (XML, JSON, properties). You can leverage the services in pub.security.outboundPasswords to store passwords used to access other systems in a secure manner. If you’re really ambitious, you can create an IS Adminsitrator-based facility to allow developers to manage passwords stored there. Then services would use pub.security.outboundPasswords services to get them when needed. Including, possibly, your general-purpose “getProperties” service.

HTH.

Hey Reamon,

Will defiintely use watt.server.http.jsonFormat instead of specifying this Content-Type = application/json in every request as headers

Now regarding your code. Its perfect nothing wrong with it but just fail to understand the last part from looking at it

IData document = IDataFactory.create();
** final IDataCursor dc = document.getCursor();**
** props.forEach((key, value) → {**
** IDataUtil.put(dc, (String)key, (String)value);**
** });**

foreach loop?

I wanted to keep this precise and simple and when I say precise just for .properties file reading.
because .properties is current/popular.
XML is becoming stale and antiquated. So I dont want to mix the two.

For XML I am using this approach

Then I am creating a custom Java service and grab the whole xml document and grab whatever value I want to grab. An example of xml would be something like Logback in spring boot.

As for what you mention on validation, I think you are right. I am just overdoing things. I will leave the validation on incoming request and expect a perfect data but a small caveat, thats not always going to be the case. We have to be a bit limber and cautious when it comes to malicious data.

Everyday there is a new challenge I am facing. Have to take this one day at a time
Thx

Actually, the Content-Type header must be specified every time. The jsonFormat URL parameter can be used by clients to override the watt.server.http.jsonFormat setting for a given call. Example: https://myserver.abc.com/rest/MyEndpoint/foo?jsonFormat=bytes will cause wM IS to place a byte array named jsonBytes var in the pipeline regardless of the watt.server.http.jsonFormat setting. Whether or not your service supports that is up to you. The Content-Type: application/json header is critical – without it, wM IS won’t provide the JSON vars to your service. (It will provide something else, but that’s a different topic for a different time.)

For the code that loads properties into an IS document: The forEach loop is iterating over all the keys in the properties object, getting the value of each and then using the properties key and value to place in the IS document. Consider this properties example:

website = https://en.wikipedia.org/
language = English
message = Welcome to \
Wikipedia!

This can be loaded from a file, put into a hard-coded string, etc. Pass any of those to the propertiesToDocument() method I shared earlier and the result will be this in the pipeline (using dots to note they are child elements of document):

document
…website
…language
…message

Then the caller can refer to and use any of those 3 variables in the document directly. Use in a MAP step, pass to a sevice in INVOKE, etc. You can define a document type that has the fields/structure that matches the fields in the properties file.

Why? The steps have converted it to an IS document. You can reference any/all of the fields from the XML in that document. There is no need to use Java to retrieve a particular value. The whole point of converting the XML to an IS document is to be able to easily access those fields. Here are some more details for what to do:

First, an example XML file to use for config.

<?xml version="1.0"?>
<properties>
<ignoreCredentials>false</ignoreCredentials>
<skipVerifySCID>000000</skipVerifySCID>
<issuerCertAliasList>
<alias>some-issuer</alias>
<alias>other-issuer_old</alias>
</issuerCertAliasList>
<trustStoreAlias>DEVTrustStore</trustStoreAlias>
<updateIdExpansionPrefix>zOS/RSU/</updateIdExpansionPrefix>
<updateOrderURITemplate>https://myHost/abcd/UpdateOrder?orderid=%/orderId%</updateOrderURITemplate>
<orderidPattern>R[A-Z0-9]{9}</orderidPattern>
</properties>

Based upon that, we have defined a document type:

With these 2 items, we call the common service we have to load the file the contents of which are now in a document variable. The “getPropertiesEx” is our service that does the getFile, xmlStringToXMLNode, etc. steps.

https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/078394206b9b02c0838a6ed610448ade70887d82.png 1.5x, https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/078394206b9b02c0838a6ed610448ade70887d82.png 2x" data-dominant-color="F4F6F7">

image985×585 34 KB

Later in that service, we simply map the fields that are needed for the processing we want to do:

https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files3/d0d2a502e28ecdd68c30b24ebf952c655848c51a_2_679x750.png 1.5x, https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/d0d2a502e28ecdd68c30b24ebf952c655848c51a.png 2x" data-dominant-color="F1F3F4">

image734×810 64.5 KB

There are no Java services in any of that. Not needed.

You’d do a similar thing if you want to use a properties file (or JSON) instead of XML. Though for properties you’d use the Java code shared to do the conversion from the properties object to the IS document.

That’s an interesting perception since properties have been around a bit longer than XML. XML has a variety of things that make it less desirable, but I don’t think age is one of them.

You could also use .ini files if the notion of sections would be useful. The Java code to parse and convert those to an IS document is a bit different but readily doable.

Don’t expect perfect data. The main point I was trying to make is that the J2EE components that you’re passing the data to should already be checking for bad/malicious data – and if they are, you don’t need to do so again.

Hey Reamon,

Let me see if I understood you correctly.
getPropertiesEx it is your own Java custom service where you are performing all these steps in one right?
steps:
getFile → xmlStringToXMLNode → xmlNodeToDocument

I meant XML is the language format of retro but now JSON is the present/now/popular.

Reamon I have another question, that I am working on.
I am using external Jar, That external Jar has some dependency FX:

Now to use any external Jars it has to be in this path for you to import and compile the code to call the Jar code
C:\SoftwareAG\IntegrationServer\lib\jars\custom

Now the problem came any dependency you have on that Jar, It has to be also included as well inside this path
C:\SoftwareAG\IntegrationServer\lib\jars\custom
otherwise when you run, it throws exception on dependencies.
So that was the solution.
But is there something I can do better. Like a pom file concept.
As you can see I have to look for these Jars on google, Download them/unzip and then copy/paste in this folder.
Now imagine this is just few dependencies I had included in the JAR but what if I had dependencies within dependencies within dependencies. Then I will have to look for all these Jars Download them/unzip and then copy/paste in this folder.
That’s tedious/tiresome work.
Do you have any experience with this.
We will convert all our projects from JAVA to Maven Project in webMethods.
Any thoughts?

Thank you very much

Yes, it is a custom service performing all those steps. It is not Java. It is a FLOW service.

There will always be a “new” and “better” markup language. Don’t be too concerned about that. Use a format for your config files that makes sense for your environment/team. Properties are fine. JSON is fine. XML is fine. .ini files are fine. Storing in a DB is fine. For any of them the concept is the same: store the settings in a file/table(s), load them into an IS document to be used. From a service perspective that uses the config in the IS document, it does not know nor care what the underlying storage format is.

Again, lot’s of things to criticize about XML. Age and “retro” should not be one of them.

The primary suggestion I have regarding external libraries – be very conservative. Don’t try to use wM IS the same way you would code/use JBoss, WLS, WAS, etc. The run-time environment is Java but this is not a Java development platform. One can “drop down” to Java to create focused utility methods but it is not intended for “significant” Java development. Do that elsewhere if it is needed. Don’t try to load all the typical libraries used when creating typical Java apps. If you keep trying to use wM IS in the same way you do Java app development, you’re going to be disappointed and frustrated.

Review the About page in IS Administrator. You may be pleasantly surprised that the library you want to use is already present. Here is a short list of ours from 10.5 (trimmed to show just the significant part of the end of the path):

common/lib/ext/commons-beanutils.jar
common/lib/ext/commons-codec.jar
common/lib/ext/commons-collections.jar
common/lib/ext/commons-digester.jar
common/lib/ext/commons-discovery.jar
common/lib/ext/commons-fileupload.jar
common/lib/ext/commons-httpclient.jar
common/lib/ext/commons-io.jar
common/lib/ext/commons-jexl3.jar
common/lib/ext/commons-lang.jar
common/lib/ext/commons-lang3.jar
common/lib/ext/commons-logging.jar
common/lib/ext/jackson-annotations.jar
common/lib/ext/jackson-core.jar
common/lib/ext/jackson-databind.jar
common/lib/ext/jackson-dataformat-yaml.jar
common/lib/ext/spring-core.jar
common/lib/ext/swagger-annotations.jar
common/lib/ext/swagger-core.jar
common/lib/ext/swagger-models.jar
common/lib/ext/swagger-parser.jar

That said, there are times when libraries are needed. You’ll want to get the minimum necessary (ideally, 1 for a given function/need) to limit the potentially tedious work of importing a pile of dependencies. If you find a deep list of dependencies that end up “importing the world” you’ll likely want to look for another library with less baggage.

The number of third-party JARs we’ve used over the years is like 2. One for SSH (which has been mostly replaced by built-in facilities for SFTP now) and one for PGP encryption (BouncyCastle – which is also now part of the core set of libs). We considered adding Jackson at some point but decided against it – and now it is part of the core list too. Often times there are useful functions available in Apache Commons. But more often than not, the relatively narrow function we wanted required “import the world” so we either looked elsewhere or created our own using Apache code as a guide. And as can be seen in the above abbreviated list, many of them are now included.

Be cautious here. This indicates a potential mismatch of how to use wM IS as an integration tool (not an app dev tool) vs how Java development is done. Referencing an old post again that cautions about being too Java-focused.

Integration Server and Java - #22 by reamon

Wait Reamon I am doing the same thing right?

I am confused now?

Yes, those are the right steps for your own getPropertiesEx service (named however you’d like).

My comment was only noting that getPropertiesEx is a FLOW service, not a Java service.

Oh ok got it boss.
Another thing Reamon from that link, I saw something about Try Catch flow step, mention

image881×229 16.3 KB

That made me think. What are the most common flow steps that are being used a lot?

I am getting the hang of BRANCH as well as MAP. LOOP a little bit.
But now for the rest of them

are they really also used alot or Invoking service can full fill the need to use the rest of them?

Hi Sufran,

TRY-CATCH, TRY-FINALLY, CATCH and FINALLY have been introduced with 10.5.

The others already existed in previous versions.
REPEAT is used when there is a (fixed, but configurable via variables) maximum of interations known and a delay might be needed between iterations.
LOOP is used to process a list of data regardless of size.
SEQUENCE is used to group steps together, i.e. in combination with BRANCH, when each branch contains more than one step.
MAP is used for mapping (and transfroming) data from one structure to another.
INVOKE is used to directly cal another service regadless of type.

See Working with Designer Guide and Service Development Help Guide for details.

Regards,
Holger

Thanks Holger.
On a debugging Note. Is there any step by step guide on logback implementation for webMethods?

On google I found out I need to install
Log4J webMethods package in the webMethods Integration Server. You can download it here: http://techcommunity.softwareag.com/ecosystem/communities/public/webmethods/products/bpm_suite/codesamples/59bf91a8-b1d3-11e4-8d00-cd8d7ef22065/

Is that true?
But link aint working.

I found this link

will try this

Thx

Reamon I also wanted to add this as well. I am searching on this thoroughly
Is there any step by step guide on logback implementation for webMethods?

On google I found out I need to install
Log4J webMethods package in the webMethods Integration Server. You can download it here: http://techcommunity.softwareag.com/ecosystem/communities/public/webmethods/products/bpm_suite/codesamples/59bf91a8-b1d3-11e4-8d00-cd8d7ef22065/

Is that true?
But link aint working.

will try this.

Thx

The least used is probably REPEAT. As Holger notes, used for a fixed number of iterations. We use it mainly for auto-retry in the face of transient errors.

The TRY-CATCH… constructs are new. They are a welcome change to the nested SEQUENCE approach that is still very common. The main downside to the nested SEQUENCES was that getLastError supported only exceptions of a specific type. CATCH is more consistent.

One guideline here – use TRY-CATCH only for top-level services. In other services (e.g. services such as those in WmPublic – used by many other services) use TRY-CATCH only to if the service needs to close a resource of some sort. E.g. file, SFTP connection, etc. This is relatively rare.

Log4J should already part of the environment. Review your About page for IS and you should see it.

I’m going to sound like a broken record here – avoid doing things as though IS is a Java app server.

We went down the path years ago of incorporating Log4J. We ended up abandoning it. Ended up not being needed. (Side note: I recall long ago when Log4J first became available and how easy it was to use. Like many things in our industry, it grew into an over-engineered monstrosity. And multiple logging frameworks evolved into nonsense like SLF4J to abstract all the logging abstractions. Ugh.)

My advice – don’t add log4j to your environment. You likely do not need it. Things we still do:

• Rely on audit logging of the top-level services and use pub.prt.log:logActivityMessages within services whenever we want to log something, primarily IDs of the documents being processed. Then we use MWS to review/search the audit logs to find activity. Mainly “what happened to XYZ.” This is our primary technique. Refer to the wM docs describing the audit log facility.

• Solution-specific logging. We have a couple of utility services implemented on top of Integration Server logging components. One is a (Java service) wrapper for com.wm.app.b2b.server.ServerAPI.getLogStream. This is what IS uses internally to write and rotate stats.log.Then we have another Java service to call to write data to the log. Provides an easy way to create log that auto-rotates at midnight. We don’t use this much.

IMO, log4j/logback is unnecessary in wM IS. Learn about audit logging instead.

Hey Reamon,

I accomplished half way with LOG4J2, I can see info on console but its not writing to the files or even creating the file/folder. I went through hell to figure out the configuration part but still half way to go. The only level is working is INFO. DEBUG nope. I accomplished the half way approach using External Jar way.

Regarding ServerAPI.getLogStream, I saw in some post you mention that.
Is this can be inside Java service or can it invoke in flow service?

Thx

One can only invoke Java object methods using Java services. To use such components FLOW you would write “wrapper” services to expose them via the IS service construct. Here are the 2 Java services that wrap the methods of interest.

// Wrapper service for com.wm.app.b2b.server.ServerAPI.getLogStream.
// This is what IS uses internally to write and rotate stats.log.

// Opens a log file in the server's default logs directory. 
// This should only be used to obtain handles to new log files for 
// solution-specific logging purposes. By convention, the name should 
// be something like "myapp.log".

// The returned stream is intended to kept open for the life of the
// JVM (don't open a whole bunch of these).

// This service maintains a hash table of log streams that have
// been opened keyed by the logfile parameter. If the log has 
// been previously opened, the existing stream is returned and 
// the "newLog" is set to "false". If the log is opened for the
// first time a new stream is returned and the output parameter
// "newLog" is set to "true".

// The closeLogStream service can be used to force a stream closed
// but this shouldn't normally be needed (intended to aid 
// development and testing).
public static final void getLogStream(IData pipeline) throws ServiceException {
IDataCursor idc = pipeline.getCursor();
String logfile = IDataUtil.getString(idc, "logfile");

boolean newLog = false;
com.wm.app.b2b.server.LogOutputStream os = 
(com.wm.app.b2b.server.LogOutputStream)openLogFiles.get(logfile); // get existing stream for this filename, if present

if(os == null)
{
os = com.wm.app.b2b.server.ServerAPI.getLogStream(logfile);
openLogFiles.put(logfile, os); // keep track of this stream for future use
newLog = true;
}

IDataUtil.put(idc, "logStream", os);
IDataUtil.put(idc, "newLog", (newLog+""));
idc.destroy();
}

// Writes the specified message to the specified stream. 
// When writing to the stream, the current date and time
// are written first, followed by the message, and terminated
// by the line.separator string from the system properties.
public static final void writeLogStream(IData pipeline) throws ServiceException {
IDataCursor idc = pipeline.getCursor();
com.wm.app.b2b.server.LogOutputStream os = 
(com.wm.app.b2b.server.LogOutputStream)IDataUtil.get(idc, "logStream");
String message = IDataUtil.getString(idc, "message");
idc.destroy();

os.write(message);
}

// Closes the stream associated with the specified
// logfile and removes the logfile from the list
// of open files. This isn't intended for normal
// operation and is provided primarily to help
// with development/testing of these log services.
public static final void closeLogStream(IData pipeline) throws ServiceException {
IDataCursor idc = pipeline.getCursor();
String logfile = IDataUtil.getString(idc, "logfile");
idc.destroy();

com.wm.app.b2b.server.LogOutputStream os = 
(com.wm.app.b2b.server.LogOutputStream)openLogFiles.get(logfile);

if(os != null)
{
openLogFiles.remove(logfile); // stop tracking this one
os.close();
}
}

// --- <<IS-BEGIN-SHARED-SOURCE-AREA>> ---
private static final java.util.Hashtable openLogFiles = new java.util.Hashtable();

// --- <<IS-END-SHARED-SOURCE-AREA>> ---

Note that these 3 services are using com.wm.app.b2b.server.LogOutputStream, a class provided by wM IS. You may notice the class name as com.wm.app.b2b.server – you may know that the original name of wM Integration Server was wM B2B Server. That’s why the class has that name.

With the above, you can have a FLOW service anywhere that calls “getLogStream” service first, passing a filename. Then pass the returned logStream variable and a string message to “writeLogStream” to write to that log file. The class used will take care the formatting of the entry and the automatic file rotation. Assuming you’ve started with a “public package” of your own, much like WmPublic provides general-purpose services, you can put these in your package and use them wherever desired.

HTH.

Thx Reamon. I will try this. It seems more limber. Got to try this.
Thx much

Hey Reamon,

Unfortunately Reamon team wants to go with either LOG4J2 or LOGBACK. Since flexibility is a bit tight on this. I have to somehow make this work. Like I mention I was half and half with extrenal Jar and in built Log4J2. I tried all day yesterday but it wouldnt write to the file. Not sure what exactly I am missing. More to come on that. Stay Tune

Thx

Wiseman speaketh.

Easiest way to grab such value is actually to capture output structure of getTransportInfo, create a document with that output structure and map getTranportInfo output to a reference of the structure. Once you do that - you can visually map specific fields without any looping or java required.