Hi,

My requirement is to watch the DB table, if any insert operation takes place send the newly inserted data in the XML document, calling the webService provided
by target client and pushing the data to them.

My thought is creating a insert DB notifictaion, poll the DB table and if any insert operation takes place then reading inserted data, preparing one xml document
and pushing it to the target client by SOAP webService ( I am the consumer of the webService ). Imagine due to some reason if I am unable to hit the target server
then how can I preserve the data which I got from the DB notification and how can I disable the notification to poll the DB for a few mins till the target Server
becomes available ( My thought is using transient error handling technique to check target Server
and if not reachable then disable the notification ).

Kindly shed some light to step ahead further.

Seems a typical use case for JDBC Adapter Notification.
in IS, try to create a: Adapter Notification
Select JDBC Adapter, Select Insert Notification, Select your connection name (if you haven’t create one, create it first).

For more details, check JDBC Adapter Installation and User’s Guide.

Thanks wang for your reply.

I will take care of creating notification, mapping those fields to xml document. After this when I hit the target server to push data and if client (target server)doesn’t able to accept then how can I store/preserve this xml data. I need some efficient way to handle this situation. Kindly share any thoughts on this.

Hi,

My 2 cents:
Keep a unique id for each transaction with the help of UUID, when you are having issues with Target, convert the XML Doc to xmlString, store it in DB, disable the DB notifcation to stop polling till you are able to make a hand shake with Target Server. Once the climate becomes normal,first you read data from db which was stored when the Target was having issues, once done then enable the notification and keep going.

Thanks,

Did you explore basic notification instead of insert db notification. There is a difference that will suite your requirement.

Thanks a lot to everyone.

Mahesh, Can you more details on what you want to tell. If possible can you give some rough design …

Hi Al,

Now I have n number of clients whose details i am getting as part of one of the field’s of the XML message. Let’s say
client1 is up,
client2 is down,
client3 is up,
client4 is up,
client6 is down,
…
…
client n is up.

If I disable the scheduler just because of one client is down then other clients get impacted but I want to stop the flow
towards that client which is down and rest of the flow to other clients which are up should continue. Kindly help me
how to handle this situation.

I am really thankful to all for your time.

This is a classic pub/sub scenario. Simply publish your message to the bus (i.e. Broker or UM) and have a subscriber/trigger per client. By default, your notification messages should already be published.

Percio

Thanks Percio for your inputs.

But if any new client is getting added then we have to add a new trigger and new service to process the document. Any other optimal solution which doesn’t make me to write any additional code though I have to support a new client.

Thanks,

Sure. There are multiple ways to skin a cat though, so here’s one possible solution:

First, create a publishable document that contains the data elements to be sent to the client + some client specific fields (e.g. URL, username, and password – or perhaps just a client ID that can be used to look up the other info from a config file)

Then implement something like this:

1. Receive the data from the insert notification

2. Retrieve the list of clients from a config file

3. For each client: map the data from the notification + the client info into the publishable document above and publish it

4. Use a concurrent trigger (with at least as many threads as potential clients) to subscribe to the message

5. The trigger service then sends the request to the appropriate client using the information in the document

This approach gives you one trigger for all the clients. It assumes though that the requests can be received out of sequence (I don’t know enough about the data or how often it’s generated to know whether that’s a problem).

Please note that you could achieve parallelism without using pub/sub, but with pub/sub you can take advantage of the trigger retry mechanism in case the client is offline.

If it’s an option, TN could also fit nicely into the picture.

Percio

By the way, I didn’t mention it before because it may have seemed obvious from my original comment, but for the sake of completeness:

Another approach is for you to publish the messages to a bus to which your clients have access. Your clients could then subscribe the messages directly from there, for example, using JMS. This would give you the most flexibility in terms of on-boarding/off-boarding new clients with very little effort since your application would always send the data to one place regardless (i.e. the bus).

This may require a shift in mindset though because, although many customers tend to use pub/sub for internal communication, it is not as commonly used for external integrations.

Percio

Thanks percio for your inputs. Let me try to implement.

Hi,

I tried to implement transient error handling mechanism for Guaranteed publishable doc.

I have configured the trigger to handle Transient Error Handling with retry failure as Suspend and retry later option.
I have also configured the retry intervals and max retry attempts.

The throwForRetry is also present in the catch block for the service that gets invoked by the trigger.
But I can see, trigger is not getting disabled.

In the subscribing service trying to invoke the webService, so as part of Monitoring service used pub.client.http( http:servername:port) and based on output setting isAvailable flag. But i can see trigger is not getting suspended. PFA snaps.

Please need help …

It’s not intuitive, especially for developers coming from the Java world, but the throwExceptionForRetry step needs to be outside of the catch block. Typically, in the catch block, you will determine whether you want to retry the message based on the error. If so, then you set a flag that gets evaluated by a BRANCH statement outside the catch block. I’m attaching a screenshot with a sample.

Percio

https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/ecba3bb85ca17e637ae8ad29b1fbe85dbab3c3a7.jpeg 1.5x, https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/ecba3bb85ca17e637ae8ad29b1fbe85dbab3c3a7.jpeg 2x" data-dominant-color="F1F1F2">

triggerRetrySample.jpg763×210 126 KB

Thanks percio for your help, let me try.

I tried but its not suspending trigger. I am on wM 8.0.1.0 version. Any IS extended settings needed ? Kindly suggest any inputs.

https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/6fb12618b8182f461d6cee011315f328fbedc98e.jpeg 1.5x, https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files/6fb12618b8182f461d6cee011315f328fbedc98e.jpeg 2x" data-dominant-color="F9F9F9">

Subscribring.jpg819×460 66.4 KB

Anil,

First thing, you’ve used the terms “disabled” and “suspended” interchangeably so I just want to make sure: how are you checking to see if the trigger is getting suspended?

Disabling a trigger and suspending it are two different things. It’s not uncommon for people to accidentally switch these terms but it’s important to understand the difference. After the retries get exhausted, the trigger’s document processing will be suspended but the trigger remains enabled. To see whether the trigger has been suspended, please go to the Trigger Management page on the IS Admin Console.

Assuming for now that the trigger is indeed not getting suspended, here are two general thoughts based on your code:

1. You may be mixing up a couple of ideas. You do NOT have to call the monitoring service from your trigger/processing service. I suppose you could but that wasn’t the intention. The IS will start calling the monitoring service once your trigger gets suspended. As soon as the monitoring service returns isAvailable = true, the IS will automatically resume your trigger.

2. Your BRANCH statement is currently problematic. According to your BRANCH statement, your service will always end in an exception because you’re using a $default label on the EXIT $flow and signal FAILURE step.

Now… just to make sure we’re not getting ahead of ourselves, let’s do this:

1. Change your trigger so that it retries 0 times
2. Remove the monitoring service from your trigger settings

This will ensure that once you call throwExceptionForRetry, your trigger will get suspended right-away and it will remain suspended.

With this done, go ahead and change your processing service so that all it does is call throwExceptionForRetry. If you want, you can add a call to debugLog or tracePipeline in the beginning so you can easily check from the log whether your service was executed.

With this in place, your trigger should get suspended no matter what. So, go ahead and publish a document. Did it get suspended? If so, then you can now start adding your logic back in and testing it as you move along. If the trigger stops behaving as you would expect it to, it should then be easier for you to pinpoint what code or configuration change caused the issue.

Good luck,
Percio

Sorry for my wording b/w suspend, disabled. I suppose to say,Suspended.

I can see trigger is getting suspended and auto enabled when the target resource is avail.

My Next question is, when the target resource is down, trigger is in suspended status and publisher is keep publishing docs (published 50 docs ). Till the time resource gets available where the published docs gets stored if the publishable doc is of Guaranteed and we are using native broker. I checked in MWS ->Administration → Messaging ->Broker Servers → Doc Types → found my doc but didn’t able to figure out how many docs are been queued in Broker because of target server unavailability. I am using suspend&retry mechanism of trigger which I stated earlier.

Please help me on this part.

Eventually I found those docs under Clients.

Thanks Percio & others for help & Time.

Hi,

But if I don’t use the concept of Suspend And Retry Latter mechanism, how Broker make sure that failed transactions should be get processed till they successful ?

Any thoughts on this …

Hi,

But if I don’t use the concept of Suspend And Retry Latter mechanism, how Broker make sure that failed transactions should be get processed till they successful ?

Any thoughts on this …

Hi,
I have one more thing to check. In the current case as the target is a db I have used sql query to see the DB availability. But lets say if I have a webService, how can we check the availability of the webService.

My thought is to use pub.clint:http, but in this if we give url as www.xxx.com, might be server is up but service or resource on specific port might be down, in this case can we use www.xxx.com:port as part of URL of http service or do we need to use any other way. Kindly help me here as well to get more clarity to handle the situation.

Anil,

If you don’t want to use “Suspend and retry later” but you still want the docs to get automatically retried via this mechanism, you can set the “Retry until” property to Successful, meaning the message will be retried until it succeeds and the trigger will not get suspended. If you go down this route, make sure you have some type of notification in place because if you accidentally throw a “retry” exception for a fatal error, the message will retry forever and it may take you a while to realize there’s a problem.

As for the availability question, the proper way would be to require (or request) your web service provider to expose a ping operation that you can hit. If these are web services that will conform to a WSDL that you will provide, then even better. You can include the ping operation in the WSDL ahead of time.

If that’s not possible, then using pub.client:http to hit the actual HTTP server would be your next best option. You could even use the HTTP method OPTIONS or HEAD to avoid pulling data from the server with a GET, for example.

Percio

Thanks a ton percio for your inputs.

My webService provider team is not giving any thing to ping a webService. Secondly, when I used pub.client:http service and what i pass as http://www.xxx.com/ or XXX Sex - Free Porn Videos at XXX.com, its giving header/status → 200. But its not correct, kindly help me here to know whether server is really up or not.

From your statement, I didn’t understand – Retry until" property. Please elaborate a bit more.

Thanks once again for your help.

Receiving an HTTP 200 OK will tell you the web server is up. If you’re still getting an HTTP-200 but the application running on that server is actually down, then simply looking at the header status code obviously will not be sufficient. You may have to parse the HTTP response being sent back in the body.

As for the “retry until” property, look at your trigger properties. It has a property called “Retry until”. If you don’t want to suspend your trigger after all retries are exhausted (in other words, if you want to keep retrying until success), then set this property to “successful” and it will retry your document until it no longer gets a “retry” exception.

The other option is to set it up so that it throws an exception after the last retry. This means that the current document will be discarded and the trigger will move on to the next one. This is a dangerous option in my opinion because you can easily lose data if you don’t take additional precautions.

Percio

Thanks Percio for your response. Issue is thought the Tomcat server is down http service is giving always 200 status which is absolutely wrong.

Kindly see “weird behavior of pub.client:http - webMethods - Software AG Tech Community & Forums” thread for more details on this.

Thanks once again for your help.

Hi All,

Once again I need your valuable inputs.

Set Trigger properties like below under transientErrorHandling section:

Retry until: Max attempts reached
Max Retry attemps: 5
Retry interval : 10 secs
On retry failure : Throw exception

In trigger flow service as part of finally block: called throwExceptionForRetry service which is been getting executed.

Called the service which published the doc. Subscribing service got invoked and when I tried to call the target end webService, webserver is down where
webService is hosted. In IS logs, can able to see that trigger service is been invoked 5 times when the target end point is unavailable. After this I can see below line
“Exception:com.wm.app.b2b.server.ISRuntimeException: Service Temporarily Unavailable while executing trigger. Rejecting Document for TriggerStore:xxx.publishDocToClient”

Does it mean IS is deleting details of the document which got failed because of transient Error. In this case what is the use of specifiying publishable doc as Guaranteed.
How can I make sure document is delivered successfully even after transienterror happend for a few hours ?

Setting the document to guaranteed ensures that the document is persisted to the disk.

Now, in general terms, guaranteed delivery of a document is only as guaranteed as you make it. If you tell webMethods to “throw an exception” on the last retry and move on to the next document, then of course, the delivery of that particular document is no longer guaranteed.

That is precisely why in my last post I said:

I rarely (if ever) use this option. If a true transient error is occurring (e.g. web service is down), then it makes no sense to throw an exception and move on to the next document because the same transient error will likely occur with the next document, and the document after that, and so on. Next thing you know, you drained your queue and processed nothing.

Percio

Did you check the Broker queue and is the doc stil there or cleared?

Percio, I changed logic. I have same WSDL for different clients. So getting client URL’s dynamically and calling the webService by passing targetEnd URL and other data dynamically each time. So in this case I have same trigger which is sending data to different clients who are sitting on different machines.

@ RMG: Can you plz gimme steps to check Broker Q in MWS. I’m not aware of this.

Thanks for your inputs.

Login MWS:

Adminitration->Messaging->Broker Servers->Servers->click on your Broker list server name->click Brokers tab->Select your specific Broker Name->click on the Clients (Click to view) and you should see all the client id’s (ie the trigger services and a column for Docs Queued will be there and from here you can browse all the stats.

HTH,
RMG

Guys,

No need to look for this document. Once you reach the last retry, if your trigger is set to “throw an exception”, it will throw the exception and acknowledge the document so it’s removed from the Broker queue. In other words, you’re saying “I’m done retrying this document, move on.”

Percio

Percio,
In this case IS should pass on NACK to broker as it could not able to succeeded. Now its Broker turn to give the doc to IS to start processing. Isn’t it ??

Thanks,

Not sure I fully understand what you have in mind. Just to make sure we’re on the same page, when it comes to guaranteed documents, the Broker never really gives a document to the IS. It simply puts a copy of the document on a trigger queue for the IS to pick up. That copy stays there until the IS acknowledges the document to the Broker, which happens in the event of one of these: 1) the document processes successfully, (2) a fatal error occurs, or (3) the maximum retry is reached on a transient error and the trigger is set to throw an exception.

What happened to you is option #3.

So, if I understood you correctly, passing a NACK to the Broker so it “gives” the document back to the IS is really no different than changing your trigger so that instead of throwing an exception, it should just keep retrying the message.

Unfortunately, you’re in a bit of a bind because you have one trigger for multiple web service providers, meaning that an unhealthy web service provider could inadvertently affect delivery to the other ones. For you, it would be great to be able to dynamically create a separate trigger/queue for each provider, but the Broker and IS are not well suited for this (unless we get into under-the-cover APIs, which I would avoid in this case). One product that would come in handy, as I stated previously, is TN, but I’m assuming you don’t have it.

With all this in mind, here are some high-level thoughts on ways you can try to mitigate this issue given your current design:

1. Increase the number of threads on the trigger. This would help minimize the issue because you would have more threads available to process messages for the healthy web services, while the other one is retrying. Of course, if you have X threads and you receive X documents destined for unhealthy web services, you’re stuck. So this would really only work if these types of issues are expected (1) to happen rarely and (2) to be resolved quickly. In this case, you’d probably want to change your trigger to keep retrying the message until it succeeds.

2. Turn on auditing on the trigger service so that you can resubmit errors from MWS. In this case, you can keep the trigger set to throw an exception on the last retry but you would then have to manually resubmit the failed document from MWS. I’m not a fan of this option because it’s manual and you could end up with lots of errors really quickly, making it a nightmare, but I figured I’d mention it just in case.

3. Add logic to your trigger service so that once you reach the last retry, you persist the document somewhere so it can be processed again later. A few options include:

a. Persisting it to the disk or to a database and adding a scheduled task to retry them periodically

b. Publishing them to a separate trigger queue. In other words, documents for web services that are misbehaving would be sent to a different trigger, which could very well execute the same exact service as the primary trigger, but they wouldn’t affect document processing for the healthy web services.

c. Re-publishing the same exact document again. This would put the document back in the same trigger queue but at the tail end, allowing other documents to process in between. If you take this option, you would probably want to add some mechanism to flag documents that were re-published X-number of times to make sure you don’t fall into an infinite loop.

These are just a few options. I’m sure we could come up with more if needed.

Good luck,
Percio

Thanks Percio for your inputs.

Let’s say, there are 100 docs exist in Broker, there are 5 threads configured in trigger and target webService is down.
At this time, if IS also goes down by any reason is the IS able to take those 100 docs and try to push to target webService ?

Thanks,

Yes.

The document is not removed from the queue until the IS acknowledges it. The IS only acknowledges the document once it’s done with it. So, if the IS goes down before acknowledging the document, when it comes back, the document will still be there in the queue waiting to be picked up again.

If anything, in this scenario, you may end up with a document being delivered more than once rather than not delivered at all. For example, consider the scenario where the IS goes down after it has successfully processed the document but before it has acknowledged it.

Percio

YES I agree with Pierco and unless the doc is ack or cleared (manually) the guaranteed docs will not get removed from the queue.

HTH,
RMG

Thanks, i tested and its working as you said. Kindly help me on below question:

I have to send around 30K documents in a day. So set a scheduler which runs every 10 mins and pushes 200 documents to publish.Now, in the trigger I set concurrent processing, thread count 10 and when transient error happens, kept a max retry count of 10 with a gap of 5 mins before each retry.

Kindly suggest is this right approach or any other changes needed in the settings.

Thanks,

PS. To complete one schedule run ( to push 200 docs ) at present, my solution takes 30-40 secs. Kindly help me on the settings which I stated earlier.

Thanks,

Is you trigger configured as Serial or Concurrent threads?

HTH,
RMG

Why do you have a scheduled task? What happened to using an adapter notification like you had mentioned in the very beginning?

Percio

Thanks RMG & Percio for your replies.

RMG : I specified trigger as concurrent thread processing.

Percio: I wrote a storedProcedure which brings documents from DB which is been called from a flow service, it is a business demand, so there is no other go for me.

RMG: As per concurrent thread processing I set threads as 10, i am sorry not to specify in my earlier post.

Thanks,

There is a stored procedure notification too. But in any case, let’s just assume that for now, you have a scheduled task of some sort, whether using notifications, scheduler, or whatever.

First, without knowing anything about your data, system, code, or anything about the web service provider’s data, system, code, etc, it’s impossible for us to tell you how to configure your application to hit a target throughput. When it comes to performance questions, the only way to find answers is to actually test and take measurements.

Having said this, if you have a hard set requirement of 30k per day, publishing 200 documents every 10 minutes will give you a max of 28,800 documents in a 24-hour period so your current approach is a bit problematic.

A couple of things:

1. Why limit yourself to 200 documents at each interval? The way I see it, once your scheduled job wakes up, it should send everything that is currently available and not go back to sleep until it’s done. If you don’t do this, it can be very easy to end up with a backlog that you can never catch up to.

2. Why limit yourself to only running every 10 minutes?

Now, let’s talk about the numbers a little. Instead of taking some wild guesses at “how many documents” and “how often”, you need to convert your requirement into something that is measurable and then you need to go perform some measurements. Let’s give it a shot:

30000 transactions per day / (24 hours per day * 60 minutes per hour * 60 seconds per minute) = 0.35 transactions per second (i.e. tps)

With this number in mind, you can now run a load test and measure your throughput to see if you’re in good shape. Another way to look at this number is that 0.35 tps means that a single transaction has about 2.86 seconds to complete. So, you could run a single transaction through end-to-end to see if you’re meeting this basic guideline. If so, then things are looking up. Of course, a true load test would be a much better indication of actual throughput due to multithreading, stress on the system, and other factors.

Hope this helps,
Percio

Anil,

Concurrent with 10 threads is good to start with but you may need to bump up to 20 or more only based on your performance matrix requirements and as always load test in this case would be good based on the comments out from Pierco and then come to conclusion on the bottle necks with your logic/IS/Broker server resources wise or any DB layer latencies etc…

HTH,
RMG

Thanks Percio & RMG for your inputs. Will perform the load test and will change the settings as needed.

Thanks,

Good deal…and pls keep this thread posted!