Can you provide additional information on what you mean by business transaction? I hope you were referrring to normal application transactions that serve a specific business need. if that's the case,both message broker and process server are capable of running parallel transaction. In Message Broker, you would achieve that using "additional flow instances" and In Process Server, the same can be achieved by increasing the number of threads associated or deploying them on multiple JVM's.
If you were referring to spanning multiple transactions from a single incoming transaction, it can be achived in both WPS & WMB(aggregation).
You might want to refer to performance metrics on WMB & WPS for details on the prformance/throughput using these tools.
With multiple tools (ESB, WMB, WPS, DataPower) providing ESB capabilities, it is often a big challenge for the IT delivery team in determing which product to use esp. when the organization has all the ESB offerings.
Hope this helps.