webMethods released a couple of white papers this August on 6.5 IS server performance and 6.5 Broker JMS performance. They are out on Advantage if you haven’t seen them.

The interesting thing (at least to me) was that the performance of the Broker was better on Windows 2003 than on Linux. Same hardware for each OS. Does that surprise anyone? It did me.

markg
http://darth.homelinux.net

Hello,
I would not be surprised that a compiled application on one architecture could outperform another. We would have to know the background processes running, and some of the options for compiling. I mean, one could be using the Intel C++ Kit and the other g++. I might post more after I actually read the papers posted. Good day.

Yemi Bedu

Links to articles Mark G referred to above.

I had a chance to look at these a little more. It’s difficult to draw to many conclusions because of the way the test were conducted. The IS portion was primarily Windows and Solaris but with a few Linux test thrown in. The broker portion was all Windows and Linux. On the IS side they did not use any of the newer Sun Sparc processors but rather some of the older slower ones. The Intel processors were more of the advanced higher performing ones. Sparc processors are never going to reach the clock speed of Intel but there are much newer ones than the ones used for the test.

I would have like to seen a more thorough test methodology where all the tests were run across all three platforms with similar configurations. I think it would have been more useful.

As I noted in a previous post, the broker performance on Linux was not very good in comparison to Windows. Have others seen this in the real world? How is the broker on Linux? Anyone running it out there?

markg
http://darth.homelinux.net

Echoing Yemi’s point about the compilers – GCC is known to be a rather dismay compiler when it comes to code efficiency, and unfortunately is the default for most Linux distributions. A 4-to-1 speed difference (Windows:Linux) someone noted just isn’t likely to be blamed on OS alone.

Some updated numbers from webMethods on JMS performance. In a nutshell, Windows is still outperforming Linux. Solaris on the 490 they used is running better than both in some tests. However they have some notes on that at the conclusion of the document.

[url=“http://advantage.webmethods.com/cgi-bin/advantage/main.jsp?w=0&s=3304956120&targChanId=-536884339&id=best_practices_detailed_product_tech_info”]http://advantage.webmethods.com/cgi-bin/advantage/main.jsp?w=0&s=3304956120&targChanId=-536884339&id=best_practices_detailed_product_tech_info[/url]

I think the core platforms for webMethods, Windows and Solaris, will probably continue to outperform the other platforms. It doesn’t mean the other platforms are slower by some inherent design flaw of the OS or hardware. It just means as a company webMethods spends most of its time on Windows and Solaris. I think that is why it is important to always know the core platform(s)of your software vendor. I think support issues (patches, maintenance releases) tend to come to the primary platforms a bit faster as well.

Just my opinion though. Everyone’s mileage will vary with platforms. It would be nice to see them run these test across all of their supported platforms with like configured hardware.

markg
http:/darth.homelinux.net

Hello,
Holy crap, you have 4TB of ram in your computer. You should have something like a 148gb ram disk and just copy everything non-persistance critical to it. Holy crap, you are running that on 32-bit Xeons.

Thank you for the real bench numbers. Where you able to get any information about how the systems seemed taxed during these runs? Like was there significant I/O or memory use at certain points in time. Also, can you tell us what JVM’s you were using and if you would be able to substitute others (sun, ibm, blackdown, ) Good day.

Yemi Bedu

We installed 6.5 into production on Sept 8th. My advice, don’t install it into production at this time if you haven’t done so already!! The performance is killing us. We have a couple of support requests already open, and will be opening another one today. 6.1 is much much faster than 6.5 in the real world. Maybe some made up performance tests show 6.5 as being better, but it’s much much worse!

Kevin,
Can you give me more details on your environment and what you are doing with the Integration platform? And what you are seeing in the way of performance issues ie where the problems are, bottlenecks etc. We are just starting our upgrade project. Thanks.

markg
http://darth.homelinux.net

Hi Mark,

Sure, here’s some info that I hope will help.

Sun Solaris 8 running on v480, 4G mem, 2CPU box. We’ve seen performance problems when multiple records are submitted to the integration server per flow. One being an LDAP update, one being a SQL read and create flat file, and a third one being incoming FTP and email out. And the numbers of records are not large by any means. For example the SQL read and flat file write is 2000 records. It was taking a few minutes with version 6.1, now with version 6.5 it has run as long as 14 hours! The slowness appears to happen logarithmic from what were seeing, or at least close to that. These flows happen to be fairly complex, so we’re still trying to figure out what is the common item in these flows that cause the problem. We don’t see anything unusual about cpu or memory usage on the server side, in fact it looks somewhat idle. But in the SQL example, we see 17 seconds between write fileIO’s on the server side, when it should be milliseconds. We’ve wondered about GC on the JVM, but the symptoms don’t suggest that, unless it’s pausing to do GC between every IO write, and the since each pass takes longer, that also does not point to GC. And you would think that a GC pause would be fairly consistent.

If you fire a new flow for each instance of data, performace appears to be okay. It’s only when a flow is required to process 1,000’s of records per invocation when we’re seeing the problems.

I can post more info as we figure out what’s the root cause of our problem.

I hope this helps! Cheers, …Kevin

Thanks kevin for the latest update.

yeh please do provide us the info on 6.5 performance issues if you find more indeed it will help for thinking of migration to Fabric6.5.

TIA,
RMG

Kevin,

Just curious, did you guys ever figure out the root cause of the 6.5 performance problems? We are currently contemplating upgrading to 6.5 as well so your inputs would much appreciated.

Thanks,
Percio

I too am interested to hear 6.5 stories…

Anyone know if webMethods has stopped using Solaris as base development platform?

I’ll let you know what I see performance wise this week. We are conducting our first real load test with 6.5. We just recently upgraded our build and test environment for one of our instances. The vast majority of the pain we have felt so far with 6.5 is the new my webMethods Server, which in my opinion was not ready for release.

My understanding (which is subject to being wrong) is Windows is their primary development platform and then Solaris with everything else coming later. Of course that may vary by development group. I still see some variety in script, patch delivery from the different groups which suggest that the development is still not completely consistent across groups although it has gotten a lot better.

So far so good with the load testing on 6.5. I have not seen any performance issues. I’m seeing some improvement in the JVM with gc, this is a result of moving from 1.3.1 to 1.4.2. I have not seen any major improvements (some minor improvements were observed) in overall throughput so far but the real test is coming later today, I’ll post the results after that. The good news at least so far is no drop off in performance, ie they didn’t break anything.

Load Test Part I
Adapters Used-MQ, JDBC, Mainframe
Volume - 6000 transactions every 3 minutes sustained for 1 hour. 120,000 total per hour. No latency observed with transactions ie the server was able to clear the 6000 transactions within the 3 minute interval prior to the next load running coming through.

I talked with a product manager at webMethods about the development environments relating to OS platforms. Solaris and Windows are still the most widely used platforms from a customer base standpoint. The developers are developing on a combination of platforms including Linux which is apparently gaining popularity within the development organization. I did express some concerns about the performance of the broker on Linux. If you read the performance whitepapers that were released back in August of last year, you will notice that Windows and Solaris outperformed Linux by a pretty significant margin. My question was the broker just ported over to Linux or was any efforts made to optimize it for the platform? I’ll let you know if I get anything back on that after they follow-up.

Mark,

Can you provide info on your IS config along with your load test results?

Thanks!!

Ray

Okay we finished up the load testing. We were able to get our stretch goal of 150K per hour. We were not able to do that with this particular integration with 6.0.1 in the past. I think a lot of it had to do with the JVM upgrade.

Here is the configuration we used:

Sun v440 4(1.2Ghz)x8GB.

A single IS instance with broker. The IS instance was configured at 512MB Max. The configuration repository was Oracle 9i 9.2.0.6.

The integration consisted of a jdbc adapter notification, MQ adapter listener and put, Mainframe Adapter transaction and multiple db lookups for data transformation and enrichement.

The data was inserted into an oracle database at the rate of 1500 transactions per 3 minutes. Those transactions were then put into the integration layer and turned into 5 distinct transactions, hence the 150k per hour.

A cool new feature in the IS Admin panel is the ability to suspend and restore individual triggers. This has a side benefit when performance tuning. It displays all of the triggers on a single page along with how many threads are configured to run (if parallel) along with how many are actually running. This makes it easier to identify where the bottlenecks are occuring. I highly recommend it if using triggers.

Mark,
May i know, for your load test, what was the transaction size?(Is it one database row , as you said you were using adapter notifications)

Also, do you have a pub/Sub Canonical Architecture for your load test case where you got 150K per hour. If so, what were your trigger concurrency settings to acheive optimal performance? I would appreciate if you could give a little more detail(though you already explained a lot) about your test scenario.

Sorry about lot of questions. I am just wondering.

Yep we used a typical pub/sub style of messaging pattern. We had 5 seperate subscription services. We also use the concept of a Canonical format. The triggers were set to concurrent. Some were set at 7 threads, some at 3. We also played around with the queue capacity and retrieve refill levels. We ended up setting those higher than the defaults, 50 on the capacity for most and 20 on the refill.

You kind of have to experiment to find the right levels.

The data was one row in the database, pretty small don’t have exact numbers but it wasn’t that big. Something around 40 fields, 10 to 15 characters on average each.

Mark, Thank you. That surely is helpful.

From the webMethods Technical Report “Hardware, JVM, and Operating System Performance Tuning Recommendations” dated November 2005 (Version 1.0)