Originally posted by: SystemAdmin

We are reading Swift messages from MQ and transform/process them individually into output files.

But I am not sure whether I should read only 1 message at a time or read them in block, as both methods have (dis)advantages, especially for error handling.

What is your experience with this?

Thanks all...
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

A lot depends on the volume of messages that you expect to appear on the queue. If the volume is relatively low (1000's per day) then you could process one message at a time in your map. This will keep error handling simple and easy to maintain.

Each instance of a map consumes a thread. Therefore, if you are expecting high volumes of data then it may be worthwhile reading in a burst of data (e.g. 100 SWIFT messages) for each map triggerred. This will reduce the number of concurrent threads you have running and speed up processing. You would have to use dynamic naming of output files via PUT or RUN commands. If errors occur, all messages will be rolled back onto the queue (depending if you use Rollback, REJECT functions, the type of errors etc). How do you then know where to restart processing from?

MQ/TX is capable of handling high volumes very well, though, of course, a lot depends on your hardware and software design.

Seeing as you have to produce one output file per SWIFT message, reading in one input message at a time would seem logical to me. Usually the simplest design is the best.
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

>How do you then know where to restart processing from?

That is the main problem which keeps me from reading them in batch...

Also is there a way to read messages from a queue and write them directly in a file (through an output card) without having to use RUN, PUT, like through a wildcard for example?
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

In the IFD, for the output card PUT definition, you can use a wildcard definition such as MyOutputName*.txt. This will then base the output name on the input name. I know this works with files but haven't tested it with queues.

With files, for example -- if your input file is named Input1.txt, your output filename will be MyOutputName1.txt.

With PUT you can control your naming and also ensure it is unique. I would definately use PUT rather than relying on the wildcard.

regards,

Tarun
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

I know it works with files, but with MQ I never tried.
Also a card could do a rollback but not PUT or RUN...
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Another way of pulling data from Q's
========================
You can also use shell script to pull data out of queues.
Like ...... this script is executed by an MQ start process or can
be scheduled....
The FILENAME parameter is passed to the script and this parameter is used as part of the file names. The script pulls data off a MQSeries Queue using the MQMSGETW utility.
Let me if you need more detail information

• Rami (rami_edi@yahoo.com)

#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

>I know it works with files, but with MQ I never tried.
Also a card could do a rollback but not PUT or RUN...
This doesn't work with queue inputs.
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: SystemAdmin

How you design the solution depends a lot on the wholistic business, audit, security, recovery and technical requirements balanced against cost, time, etc.

For example, if you have a requirement to:

• ensure you never miss a message,
• never process a message more than once and
• provide an audit trail of what was received/processed when and what the result was

You might want to have one event that simply reads X messages at a time per thread, transforms and writes the output files to a work directory with a generated unique file name (possibly leveraging the MsgID or CorrelID from the message and date/time to ms + index) then have an output card that has a list of the files and their ID's. The scope of committing this output file or table is synch'd with the scope of deleting the input messages. This list of committed files is then read and moves the files from the work directory to a committed one. If using a table you could assign a "batch number" to the group of messages processed for further audit ability. If you need further fail over protection you might actually put the detail message into a blob in the database and then update the status on the row with each step.

Another technique would be to create a single output tree that could contain the output related to multiple messages and artificially create an initiator and terminator for the group. You would then commit this file and the removal of all messages that created it at the same time. A second process could have a special tree that would be used to split this file into the individual files in a work directory and a subsequent step to move them into a ready directory.

If you further had to ensure you processed the messages in order you might want to have only a single thread running. If this caused an issue with response time for individual messages as measured from when they arrived on the queue, you would look for ways to group them such as splitting the one queue into multiple queues based on something in the message header while treating the message body as a blob, etc.

If your requirements are fairly straightforward... don't care about the order or audit, just want to convert each message into a file --- leaving the MQ settings at single is the simplest. If this meets your throughput requirements then there is no need to go further. If it doesn't, create a map that will read multiple messages and create the files using run or put (could do both in a test) in one output card and have a subsequent element rule in the same output card issue an explicit FAIL. Do the same with it being a subsequent output card that issues the FAIL. Do this with various input and output on success / on failure settings and record the results. What you want to end up with is the message left on the queue if the file is not created and removed from the queue if it is. Now you can play with the message QTY and LSN parameters and the number of threads and compare the throughput you get to the single message process.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

Thank you for your response.

In fact our requirements are that no message is processed 2 times and that they are handled in the same order they arrived on the queue.

What I am doing right now is having a single map that reads a single message at the time from the queue and creates a file in a working directory for each message, named with a timestamp to make it unique and deletes the input message on success and rollback on failure.

Another map picks the files up from the working directory one at a time and does the processing /transformation. They should be taken in the original order as this is based on the creation date I think.

I would like to read them in bulk though, but then I have the problem wih where to restart when there is a failure?

Even your second solution as to read them in bulk, create a single file in a working dir and then have another map split the messages into single files has this problem doesnt it?
If for some reason you fail in the middle of the splitting, how do you proceed? This is my main concern...

I do not really understand your first solution and which specific problems it would solve.
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: SystemAdmin

First, if you want to guarantee you process the messages in an exact order you may need to do something other that file based triggers and you wouldn't multi-thread. As soon as you multi-thread you cannot guarantee order.

This is probably not a subject that you will get an answer for in this forum because architecture/design is a series of questions, clarifications and evaluations. You may want to consider bringing a TX Technical Architect in from Ascential to help with your design.

The simplest implementation is:

• Read one message per thread
• Transform the message and create the final output (execute your transformation map through a RUN)
• Scope of map removes the message and commits the output
• No intermediate file landing required

The second solution that puts all the messages into a single file and then splits them through another map would either:

1) rely on scope settings that would not "commit" the output files (rename them) until completion - in which case you just rerun the map and clean up the tmp files in your maintenance window.
2) Create the split files in a working directory or with a different extension and have a final output card that creates a file/table with a list of all the files. If this final output card doesn't get created the files don't get moved or renamed and the original consolidated file doesn't get deleted. When you restart you will either overwrite the split files or create new ones depending on your naming logic. Either way it doesn't matter.

(This 2nd option is the same approach used in the first solution. Hope this helps.)
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

You can use wildcards in the file output only if you have a wildcard in the MQ input command lime that will get a discrete value for every message. You also have a requirement to retrieve messages in the order they are placed on the queue. If they are not committed in this order you will have a problem. You might increase performance by changing the pollwaitimemin from the default of 2000 to 15 to 45. You should also use the -EQN option so that bad data does not get rolled back to the original queue to get picked up again if the Event Server is restarted.
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: SystemAdmin

Thanks alot for the clarifications.

I have one additional question about this though:
>1) rely on scope settings that would not "commit" the output files (rename >them) until completion -

My understanding is that maps called through RUN (used here for file splitting) are discrete from the calling parent map. How would you set such a scope setting?
Also could anyone confirm that the file adapter will always grab files based on creation time?
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: SystemAdmin

Sorry, my bad. Confused a request for enhancement I had submitted with reality. (Commit won't help here.)
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: SystemAdmin

File adapter gets files in the order the OS / file listener "sees" them. No way to guarantee which file gets picked up first, especially if you are watching more than one directory. Since you use MQ, the MID is sequential I belive and you could use that to make sure everything stays in sequence; also using a DB table in between.
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender