Hi Brian,
I spent even more time on the retry problem In fact you are right that your TransitionBack step should have the retry count set correctly. It was set to 1 and this was the reason why the process went to the process-wide Error step. For the second iteration of the TransitionBack step with retry count of 1 the process would normally follow Retries Exceeded transition. As there was no transition like that the process went to Error.
It was difficult to notice the problem because I had the retry count of ToBeRetried set to 2. And in fact the step was executed 2 times. But if I set the retry count to 3 the step would still be executed only 2 times because the TransitionBack would go to Error in the second iteration.
So one should set the retry count of the TransitionBack to the same value as the ToBeRetried retry count. In this situation the process would never go to the Error step but rather follow the Retries Exceeded transition. This would also mean that you need to specify the Retries Exceeded for the TransitionBack step because you need to do so for each step with retry count > 1.
Instead of doing all this I would recommend simply use Empty step instead of the Flow step as the TransitionBack step.
Thanks Brian again for your input. Now I have full understanding of how the things work.
#Integration-Server-and-ESB#webMethods#Flow-and-Java-services