A Common Cause of "The method 'xa_end' has failed with errorCode '100'" - Asynchronous Puts

Tom_Leend |Jan 8 2016 Updated

In the IBM MQ Java Messaging Level 3 Support Team, we regularly see customers reporting issues where WebSphere Application Server is trying to end and commit a transaction involving MQ, and reports the following error:

...
Caused by: javax.transaction.RollbackException
    at com.ibm.tx.jta.impl.TransactionImpl.stage3CommitProcessing(...)
    at com.ibm.tx.jta.impl.TransactionImpl.processCommit(...)
    at com.ibm.tx.jta.impl.TransactionImpl.commit(...)
...
Caused by: javax.transaction.xa.XAException: The method 'xa_end' has failed with errorCode '100'.
    at com.ibm.mq.jmqi.JmqiXAResource.end(...)
    at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.end(...)
    at com.ibm.ejs.j2c.XATransactionWrapper.end(...)
    at com.ibm.ws.Transaction.JTA.JTAResourceBase.end(...)
...

There are a number of root causes as to why an XA End call might fail with the XA return code XA_RBBASE (100).  The following IBM Technote describes one of the most common reasons, which is related to the Aged Timeout setting on a JNDI defined JMS Connection Factory in WSAS:

http://www-01.ibm.com/support/docview.wss?uid=swg21508472

Another common cause of the exception:

  javax.transaction.xa.XAException:  The method 'xa_end' has failed with errorCode '100'.      

is related to the use of the MQ "Asynchronous Put" functionality. When this is turned on, any failures to put a message to a queue or topic are not reported immediately. Failures are only reported when the the unit of work that the messages were put under is ended - when the Transaction Manager component attempts to end and commit the transaction, the queue manager (which is acting as a resource manager in the transaction) reports that the transaction needs to be rolled back instead.

To illustrate this, imagine an IBM MQ classes for JMS application that performs the following operations inside an XA transaction:

• Looks up a JMS QueueConnectionFactory from the JNDI repository
• Calls JMSQueueConnectionFactory.createConnection()
• Calls JMSConnection.createSession()  
• Looks up a JMS Destination from the JNDI repository, which has been configured to allow asynchronous puts (property PUTASYNCALLOWED, short name: PAALD)  
• Calls JMSSession.createProducer(javax.jms.Destination)
• Calls JMSSession.createTextMessage("Hello, world!")  
• Calls JMSMessageProducer.send(javax.jms.Message) 
• Calls JMSMessageProducer.close()
• Calls JMSSession.close()
• Calls JMSConnection.close()

Now, suppose that the queue the application was trying to put the message to was full. Because the asynchronous put functionality is being used, the call to JMSMessageProducer.send(javax.jms.Message) does not throw an exception and appears to work successfully. However, the message was not actually put to the queue, as the queue was full.

After performing these actions, the Transaction Manager issues a XATransaction.end() call to end the transaction. At this point, the Transaction Manager thinks that the message was successfully sent to the queue, and so is getting ready to commit the unit of work. The end() is flowed to the queue manager, which responds with error code 100 - the message could not be put to the queue, so the transaction should be rolled back rather than committed.  This results in the Java exception noted above being written to the application server's SystemOut.log file. 

The XA_RBBASE ('100') error code corresponds to the generic reason code "the roll back was caused by an unspecified reason".  As no other exceptions are reported during the application's processing, due to the asynchronous put functionality, it does not provide us with much information about the cause of the failure.    

Now, normally, when an application sends messages to a destination, the IBM MQ classes for JMS waits for the queue manager to confirm that it has processed the request before returning control to the application.  When using asynchronous put functionality, however, the queue manager does not return the success or failure of each call and the IBM MQ class for JMS returns control to the application immediately.  In the latter case, the point at which the application is made aware that the message put request failed is when the transaction under which it was performed is closed.

Many IBM MQ customers have reported such failures when attempting to put large messages; typically messages greater than 4 MB in size.  This is often because the maximum message size configured on the destination queue, queue manager and the server-connection channel (if connecting using the CLIENT transport mode), has not been increased from the default value of 4 MB. 

When not using asynchronous puts and calling the method JMSMessageProducer.send(javax.jms.Message) to send a message larger than the maximum allowed by the MQ queue, the send request would fail and the JMSException:

 JMSWMQ2007:  Failed to send a message to destination '<queue_name>'.  JMS attempted to perform an MQPUT or MQPUT1;  however WebSphere MQ reported an error. Use the linked exception to determine the cause of this error.; nested exception is:     

 JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2030' ('MQRC_MSG_TOO_BIG_FOR_Q').

would be thrown to the application to subsequently handle. 

This would give the application developer and IBM MQ administrator immediate feedback on the issue preventing the message put from completing.  This underlying problem then becomes apparent and action can be taken to address it; i.e., in the above example case, ensuring that the queue in use has been configured with a sufficiently large maximum message size to accommodate the size of messages that the application generates.

The asynchronous put function, as well as preventing errors in messaging operations being immediately reported to applications, offers little benefit in all but a few circumstances.  In the case of sending one or two messages as part of a transactional unit of work, the use of asynchronous puts would typically be discouraged.  

In WSAS, asynchronous puts can be disabled on the JNDI entry for the JMS Destination.  Using the WSAS Administration Console, the option can be located like so:

  Resources -> JMS -> Queues -> [select entry] -> Advanced properties   
                                                                        
Then navigate to the section titled:                        
                                                                        
  "Optimizations"                                                       
                                                                        
  "Asynchronously send messages to the queue manager"                   
  --> set to "As per queue definition" or "No"                          

We hope this has given you some insight into a very common cause the exception:

javax.transaction.xa.XAException:  The method 'xa_end' has failed with errorCode '100'.

that is logged to the WSAS SystemOut.log file.  If you observe this error on your WSAS system, we recommend you disable asynchronous put functionality unless you have a specific requirement for using it.

For reference, after upgrading from pre to post WSAS version 7, asynchronous put functionality is enabled by default; without the administrator being aware.  This issue is documented in the following Technote:
                                                                        
  http://www-01.ibm.com/support/docview.wss?uid=swg21660175