Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Lower lokas of Samsara. I mean there should be hardcoded locks in firmware. I believe that the fix exists, just take a  try.

----------------------------------------------------

Алексей Марков

Администратор БД

АШАН ТЕХ

al.markov@auchan.ru

Тел.:  +7 (800) 700 58 00 доб. 79224016

IP:79224016 FMTN: *79224016

Моб.: +7 (903) 665 13 18

чт, 12 сент. 2024 г. в 23:14, jack smith via IBM TechXchange Community <Mail@connectedcommunity.org>:

And more than 10 years later I ran into the same issue as described here. Dealing with folders with lots of small files is ridiculously slow. As an example, my test case with logging enabled took 116 seconds. Without the log it took 4.1 seconds. Of course having a log takes its toll but this is way too much. Even more so since doing the very same thing with SUSE and XFS (with journaling) on the same machine with the same disk doesn't even take 3 seconds. So XFS with logging is even faster than JFS2 without logging? That seems strange. I found similar reports on other websites but, just like here, none had a proper solution. So which buttons do I have to push to get reasonable speeds with JFS2 and logging? ------------------------------ jack smith
Original Message: Sent: Thu November 07, 2013 10:06 AM

And more than 10 years later I ran into the same issue as described here. Dealing with folders with lots of small files is ridiculously slow. As an example, my test case with logging enabled took 116 seconds. Without the log it took 4.1 seconds.
Of course having a log takes its toll but this is way too much. Even more so since doing the very same thing with SUSE and XFS (with journaling) on the same machine with the same disk doesn't even take 3 seconds. So XFS with logging is even faster than JFS2 without logging? That seems strange.

I found similar reports on other websites but, just like here, none had a proper solution.

So which buttons do I have to push to get reasonable speeds with JFS2 and logging?

And more than 10 years later I ran into the same issue as described here. Dealing with folders with lots of small files is ridiculously slow. As an example, my test case with logging enabled took 116 seconds. Without the log it took 4.1 seconds.
Of course having a log takes its toll but this is way too much. Even more so since doing the very same thing with SUSE and XFS (with journaling) on the same machine with the same disk doesn't even take 3 seconds. So XFS with logging is even faster than JFS2 without logging? That seems strange.

I found similar reports on other websites but, just like here, none had a proper solution.

So which buttons do I have to push to get reasonable speeds with JFS2 and logging?

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

And more than 10 years later I ran into the same issue as described here. Dealing with folders with lots of small files is ridiculously slow. As an example, my test case with logging enabled took 116 seconds. Without the log it took 4.1 seconds.
Of course having a log takes its toll but this is way too much. Even more so since doing the very same thing with SUSE and XFS (with journaling) on the same machine with the same disk doesn't even take 3 seconds. So XFS with logging is even faster than JFS2 without logging? That seems strange.

I found similar reports on other websites but, just like here, none had a proper solution.

So which buttons do I have to push to get reasonable speeds with JFS2 and logging?

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

And more than 10 years later I ran into the same issue as described here. Dealing with folders with lots of small files is ridiculously slow. As an example, my test case with logging enabled took 116 seconds. Without the log it took 4.1 seconds.
Of course having a log takes its toll but this is way too much. Even more so since doing the very same thing with SUSE and XFS (with journaling) on the same machine with the same disk doesn't even take 3 seconds. So XFS with logging is even faster than JFS2 without logging? That seems strange.

I found similar reports on other websites but, just like here, none had a proper solution.

So which buttons do I have to push to get reasonable speeds with JFS2 and logging?

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

> I assume the JFS2 file system was not mounted with any special mount flags
I always use "noatime" and for the tests without logging I used "log=NULL" in addition.

> How was the FS created?
mklv -y testlv testvg 100
crfs -v jfs2 -d testlv -m /mnt/test -A no -p rw -a options=noatime -a agblksize=4096 -a isnapshot=no

> Did you evaluate iostat data for "service queue full",  IO service times in the case with logging?
Not yet.

> You are running  AIX 6.1 which is ancient history
Sorry, I should have mentioned that I ran into the same problem but with current AIX versions. I encountered the same problem with: 7.2.5.7, 7.3.2.1 and 7.3.2.2.
The machine in question is a POWER8 and I used a single, original IBM 146gb 15k sas hdd.

> I recommend you upgrade AIX 7.4
7.4? I don't think that's available yet.

> Is the Linux and XFS you are comparing AIX too, the same age?
Yes, both the most recent versions.

> How many is "a ton of small files" ?    Millions?
My test case had only ~10000.

> Your SAS disk is only going to manage 200 Disk I/O per second and is seems to be the limiting factor.
Well, XFS doesn't seem to think so given the results I had with that.

> Normal tuning would involve added extra disks to spread the I/O
I did that as well. I added an additional disk to the VG and used that for the log. It brought the duration down from 116 seconds to 98 seconds. Still a joke compared to the 4.1 seconds without logging and the XFS results.

> Let me know if you would like a copy of the paper. I am happy to share it.
Oh definitely!

To broaden the overall picture I ran the same test on different machines. One on OSX with a traditional hdd formatted with HFS+ with journaling and the result was 4.7 seconds.
And one more on Solaris with ZFS and a traditional hdd as well. The first run with sync set to "standard" took 4.3 seconds and with sync disabled 3.6. With sync set to "always" it even surpassed JFS2's log and took 135 seconds.
Considering all results, something around 4 seconds seems to be reasonable.

Apparently ZFS' forced sync and JFS2's log cause heavy strain on the disks. The others seem to handle logging/journaling very differently. At least as far as the results go.

Worth a note is that the Solaris docs explicitly say that setting sync to "disabled" does not increase the risk of corrupting the pool. JFS2 without logging on the other hand required an fsck after just 3 reboots.

A 15K RPM disk has an average rotational latency of 2ms, there's no way to create 10000 files in 4 seconds and maintain the required semantics.

What you're comparing is JFS2, where the file creation is committed to disk before you get control back and XFS in a mode where you get control back before anything is written to the disk.  (And yes, JFS2 is bad for micro-spooling)

> I assume the JFS2 file system was not mounted with any special mount flags
I always use "noatime" and for the tests without logging I used "log=NULL" in addition.

> How was the FS created?
mklv -y testlv testvg 100
crfs -v jfs2 -d testlv -m /mnt/test -A no -p rw -a options=noatime -a agblksize=4096 -a isnapshot=no

> Did you evaluate iostat data for "service queue full",  IO service times in the case with logging?
Not yet.

> You are running  AIX 6.1 which is ancient history
Sorry, I should have mentioned that I ran into the same problem but with current AIX versions. I encountered the same problem with: 7.2.5.7, 7.3.2.1 and 7.3.2.2.
The machine in question is a POWER8 and I used a single, original IBM 146gb 15k sas hdd.

> I recommend you upgrade AIX 7.4
7.4? I don't think that's available yet.

> Is the Linux and XFS you are comparing AIX too, the same age?
Yes, both the most recent versions.

> How many is "a ton of small files" ?    Millions?
My test case had only ~10000.

> Your SAS disk is only going to manage 200 Disk I/O per second and is seems to be the limiting factor.
Well, XFS doesn't seem to think so given the results I had with that.

> Normal tuning would involve added extra disks to spread the I/O
I did that as well. I added an additional disk to the VG and used that for the log. It brought the duration down from 116 seconds to 98 seconds. Still a joke compared to the 4.1 seconds without logging and the XFS results.

> Let me know if you would like a copy of the paper. I am happy to share it.
Oh definitely!

To broaden the overall picture I ran the same test on different machines. One on OSX with a traditional hdd formatted with HFS+ with journaling and the result was 4.7 seconds.
And one more on Solaris with ZFS and a traditional hdd as well. The first run with sync set to "standard" took 4.3 seconds and with sync disabled 3.6. With sync set to "always" it even surpassed JFS2's log and took 135 seconds.
Considering all results, something around 4 seconds seems to be reasonable.

Apparently ZFS' forced sync and JFS2's log cause heavy strain on the disks. The others seem to handle logging/journaling very differently. At least as far as the results go.

Worth a note is that the Solaris docs explicitly say that setting sync to "disabled" does not increase the risk of corrupting the pool. JFS2 without logging on the other hand required an fsck after just 3 reboots.

In the meantime I tried the following:

ioo -p -o j2_dynamicBufferPreallocation=128
ioo -p -o j2_maxRandomWrite=32
ioo -p -o j2_nBufferPerPagerDevice=2048
ioo -p -o j2_nPagesPerRBNACluster=512
ioo -p -o j2_nRandomCluster=4
ioo -p -o numfsbufs=1568
vmo -p -o maxperm%=90 -o maxclient%=90 -o minperm%=3

None had any noticeable effect on the log speed.

In the meantime I tried the following:

ioo -p -o j2_dynamicBufferPreallocation=128
ioo -p -o j2_maxRandomWrite=32
ioo -p -o j2_nBufferPerPagerDevice=2048
ioo -p -o j2_nPagesPerRBNACluster=512
ioo -p -o j2_nRandomCluster=4
ioo -p -o numfsbufs=1568
vmo -p -o maxperm%=90 -o maxclient%=90 -o minperm%=3

None had any noticeable effect on the log speed.

> Is your Linux machine using SAN disk or local SAS disk?
As mentioned in my first post already: "same machine with the same disk"

> Have you tried using an inline log for the jfs2 filesystem?
Yes, in fact for all tries except the one time with a separate log lv as mentioned already as well.

> Using both of these options will a) spread the work evenly over multiple disks b) INLINE log will spread the log over multiple disks to improve performance of the log.
Of course using more disks will speed things up but that's neither the point nor the solution. The point is that in its default form the log, inline or on a separate lv, slows things down massively. Much more than any other filesystem I've ever used.

So my simple question is: can the JFS2 log be tuned or configured in a way to reduce the speed impact? To be usable in practice it should at least come near the performance levels of other filesystems.

> What you're comparing is JFS2, where the file creation is committed to disk before you get control back and XFS in a mode where you get control back before anything is written to the disk
Well, after the command was done there was no disk led activity anymore.

> Is your Linux machine using SAN disk or local SAS disk?
As mentioned in my first post already: "same machine with the same disk"

> Have you tried using an inline log for the jfs2 filesystem?
Yes, in fact for all tries except the one time with a separate log lv as mentioned already as well.

> Using both of these options will a) spread the work evenly over multiple disks b) INLINE log will spread the log over multiple disks to improve performance of the log.
Of course using more disks will speed things up but that's neither the point nor the solution. The point is that in its default form the log, inline or on a separate lv, slows things down massively. Much more than any other filesystem I've ever used.

So my simple question is: can the JFS2 log be tuned or configured in a way to reduce the speed impact? To be usable in practice it should at least come near the performance levels of other filesystems.

> What you're comparing is JFS2, where the file creation is committed to disk before you get control back and XFS in a mode where you get control back before anything is written to the disk
Well, after the command was done there was no disk led activity anymore.

> Is your Linux machine using SAN disk or local SAS disk?
As mentioned in my first post already: "same machine with the same disk"

> Have you tried using an inline log for the jfs2 filesystem?
Yes, in fact for all tries except the one time with a separate log lv as mentioned already as well.

> Using both of these options will a) spread the work evenly over multiple disks b) INLINE log will spread the log over multiple disks to improve performance of the log.
Of course using more disks will speed things up but that's neither the point nor the solution. The point is that in its default form the log, inline or on a separate lv, slows things down massively. Much more than any other filesystem I've ever used.

So my simple question is: can the JFS2 log be tuned or configured in a way to reduce the speed impact? To be usable in practice it should at least come near the performance levels of other filesystems.

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

> if you could repeat your test on XFS with the delayed logging disabled
It seems that's not possible anymore. According to https://linux-xfs.oss.sgi.narkive.com/TPH8cSZ0/patch-01-27-xfs-update-mount-options-documentation the nodelaylog mount option has been removed more than 10 years ago.
But even if that would result in more similar results, it would still not solve the problem. Unless JFS2's log has something similar to XFS' delayed logging that can be enabled optionally.

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

> but I'm pointing out where the bottleneck is for this case
Oh I see where the bottleneck is. The question is what other filesystems do to seemingly ignore that. And in turn if and how JFS2 can be configured in a similar way.

I wouldn't mind if it was twice as slow as others. Even 3 or 4 times slower would still be okay. But 29 times slower compared to without logging and other filesystems even with logging enabled is just not usable.

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

> but I'm pointing out where the bottleneck is for this case
Oh I see where the bottleneck is. The question is what other filesystems do to seemingly ignore that. And in turn if and how JFS2 can be configured in a similar way.

I wouldn't mind if it was twice as slow as others. Even 3 or 4 times slower would still be okay. But 29 times slower compared to without logging and other filesystems even with logging enabled is just not usable.

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

> but I'm pointing out where the bottleneck is for this case
Oh I see where the bottleneck is. The question is what other filesystems do to seemingly ignore that. And in turn if and how JFS2 can be configured in a similar way.

I wouldn't mind if it was twice as slow as others. Even 3 or 4 times slower would still be okay. But 29 times slower compared to without logging and other filesystems even with logging enabled is just not usable.

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

> but I'm pointing out where the bottleneck is for this case
Oh I see where the bottleneck is. The question is what other filesystems do to seemingly ignore that. And in turn if and how JFS2 can be configured in a similar way.

I wouldn't mind if it was twice as slow as others. Even 3 or 4 times slower would still be okay. But 29 times slower compared to without logging and other filesystems even with logging enabled is just not usable.

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

> You'll have to weigh that against your application and user expectations
No, I have to weigh that against everything else that's available in the same market segment and in comparison to that JFS2's log is simply abysmal.

> Linux filesystems are often not as robust and have in the past taken shortcuts to sacrifice reliability for speed
As mentioned already, I also compared it with HFS+ and ZFS so this is not a Linux vs. AIX matter. Rather JFS vs. everything else.

> If one is suspiciously fast
Not one but all others.

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

> AIX carries workloads I would never trust to Linux
As I said twice now already: I did not only compare with Linux but also with Solaris (ZFS) and OSX (HFS+). Do I have to quote these previous posts?

> Slow enough to cause a problem?
Of course, that's the very reason why I posted in this thread in the first place.

> are there still problems with your process?
Of course, the situation hasn't changed at all. How should the problem magically be gone?

> Are you trying to frequently open lots of tar files in bulk for your application?
One of the envisaged purposes of the server in question is a fileserver. So yes, there's going to be exactly that workload all the time. If that was a one-off thing I wouldn't have asked here at all and just waited for it to finish. Whenever that might have happened.

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Considering how old this issue is, I highly doubt that IBM will suddenly start enhancing the log just because I asked nicely.

So to summarize: there are no settings which change the way the log works and neither any which impact the log in some other way. Is that correct?

------------------------------
jack smith

Original Message:
Sent: Thu November 07, 2013 10:06 AM
From: Archive User
Subject: JFS2 horrible slow

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Considering how old this issue is, I highly doubt that IBM will suddenly start enhancing the log just because I asked nicely.

So to summarize: there are no settings which change the way the log works and neither any which impact the log in some other way. Is that correct?

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Thanks for the confirmation. Obviously the log needs some sort of flash storage for competitive results.

Also since all replies only focused on Linux, maybe my edited post was not transmitted to those following by email only. So here is that part again:

"To broaden the overall picture I ran the same test on different machines. One on OSX with a traditional hdd formatted with HFS+ with journaling and the result was 4.7 seconds.
And one more on Solaris with ZFS and a traditional hdd as well. The first run with sync set to "standard" took 4.3 seconds and with sync disabled 3.6. With sync set to "always" it even surpassed JFS2's log and took 135 seconds.
Considering all results, something around 4 seconds seems to be reasonable.

Apparently ZFS' forced sync and JFS2's log cause heavy strain on the disks. The others seem to handle logging/journaling very differently. At least as far as the results go.

Worth a note is that the Solaris docs explicitly say that setting sync to "disabled" does not increase the risk of corrupting the pool."

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri

Originally posted by: XF07_Harald_Dunkel

Hi folks,

I've got 2 8231running AIX 6.1. 32GByte RAM, 2 SAS disks. No RAID. No virtual hosts. Problem: If logging is enabled, then JFS2 on a local disk is slower than a NFS connection to a remote Linux PC. My colleagues are complaining about the poor performance.

To give you some numbers:

Extracting the linux source tarball for 3.11.6 on a local JFS2 filesystem takes about 35 minutes. If I mount a remote filesystem via NFSv4 and use it for the same test, then it takes just 2 minutes. If I run the test local on the NFS server (Linux, amd64) then it takes only a few seconds, including sync.

I understand that this is a special case, writing a ton of tiny files. On daily work the poor performance doesn't show that much, but it is sufficient that nobody likes to work on the AIX hosts.

Is there something misconfigured? AFAICS the documentation says, that asynchronous IO is enabled by default. Using external or inline logging doesn't make a huge difference.

Every helpful comment would be highly appreciated.

Harri