AIX

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

So you are talking about GPFS, not JFS2? That changes really a lot (if not everything).

Backing up large GPFS filesystems is not a simple and easy task. I strongly suggest careful planning of backup strategy, taking into account creating snapshots, filesets, splitting backup into smaller tasks and splitting them between multiple GPFS nodes. GPFS tuning might be necessary, too.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Thank you everyone for the response. Networker doesn't support GPFS over AIX now in their latest version, so we are taking backup from Linux servers which took 24-25 days. However when network supported those things in the past it took approx 9-10 days with AIX node. Can we do anything at oslevel to make this job easy for backup team. Do we have any best practices or recommendation? How are we handling such things in other accounts/engagement? 

So you are talking about GPFS, not JFS2? That changes really a lot (if not everything).

Backing up large GPFS filesystems is not a simple and easy task. I strongly suggest careful planning of backup strategy, taking into account creating snapshots, filesets, splitting backup into smaller tasks and splitting them between multiple GPFS nodes. GPFS tuning might be necessary, too.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Thank you everyone for the response. Networker doesn't support GPFS over AIX now in their latest version, so we are taking backup from Linux servers which took 24-25 days. However when network supported those things in the past it took approx 9-10 days with AIX node. Can we do anything at oslevel to make this job easy for backup team. Do we have any best practices or recommendation? How are we handling such things in other accounts/engagement? 

So you are talking about GPFS, not JFS2? That changes really a lot (if not everything).

Backing up large GPFS filesystems is not a simple and easy task. I strongly suggest careful planning of backup strategy, taking into account creating snapshots, filesets, splitting backup into smaller tasks and splitting them between multiple GPFS nodes. GPFS tuning might be necessary, too.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

The way GPFS backs up to TSM is through the use of policies, which distribute the work of identifying changed files to many nodes, and then again the work of doing the actual backup.

Check mmbackup to see if you can adapt it to Networker.

As to the directory with a million files: any backup software is going to have issues with that on any operating system.
This usually has to go through a very serious conversation with the app development group.

Thank you everyone for the response. Networker doesn't support GPFS over AIX now in their latest version, so we are taking backup from Linux servers which took 24-25 days. However when network supported those things in the past it took approx 9-10 days with AIX node. Can we do anything at oslevel to make this job easy for backup team. Do we have any best practices or recommendation? How are we handling such things in other accounts/engagement? 

So you are talking about GPFS, not JFS2? That changes really a lot (if not everything).

Backing up large GPFS filesystems is not a simple and easy task. I strongly suggest careful planning of backup strategy, taking into account creating snapshots, filesets, splitting backup into smaller tasks and splitting them between multiple GPFS nodes. GPFS tuning might be necessary, too.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Thank you everyone for the response. Networker doesn't support GPFS over AIX now in their latest version, so we are taking backup from Linux servers which took 24-25 days. However when network supported those things in the past it took approx 9-10 days with AIX node. Can we do anything at oslevel to make this job easy for backup team. Do we have any best practices or recommendation? How are we handling such things in other accounts/engagement? 

So you are talking about GPFS, not JFS2? That changes really a lot (if not everything).

Backing up large GPFS filesystems is not a simple and easy task. I strongly suggest careful planning of backup strategy, taking into account creating snapshots, filesets, splitting backup into smaller tasks and splitting them between multiple GPFS nodes. GPFS tuning might be necessary, too.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

On a Large SAS Analytics server I Managed a long time ago we had 200,000+ files that totaled 24TB , and backup was always a problem . As part of the modernisation I worked with Sas Engineers to split 200,000+ files from one single filesystem , and organised files into multiple filesystems to match business units/function. I then worked with our company backup engineers ( using Netbackup not TSM in this case ) to create multiple netbackup policy's that target specific filesystems to backup . To perform the targeted approach we used the netbackup exclude/include lists feature to ensure consist backups

Prior

/sasprd

/aenv

New Layout

/sasprd/

/aenv/businessunit1

/aenv/businessunit2

/aenv/businessunit3

/aenv/businessunit4

/aenv/businessunit5

/aenv/archive

etc

# Create 5 netbackup policy's businessunit1, businessunit2, etc to target each of children filesystems of /aenva

cat /usr/openv/netbackup/exclude_list.businessunit1

/sas/prd

/aenv/businessunit2

/aenv/businessunit3

/aenv/businessunit4

/aenv/businessunit5

/aenv/archive

cat /usr/openv/netbackup/include_list.businessunit1

/aenv/businessunit1

Additional, as part of the modernisation the project – 1) Configured a dedicated highspeed Network adapter that used only for backup . 2) Setup a netbackup archive policy to prevent the system growing allow engineers to archive data , to netbackup instead of making a backup mv filea filea.notrequired .

I am not a TSM expert, is there way to using include & exclude lists to create targeted policy's in similar manor ?  It would assume that your GPFS Filesystem layout was structured  , not just  one filesystem, and you had way to archive  data offline to long term storage .

Brian Walker 

Thank you everyone for the response. Networker doesn't support GPFS over AIX now in their latest version, so we are taking backup from Linux servers which took 24-25 days. However when network supported those things in the past it took approx 9-10 days with AIX node. Can we do anything at oslevel to make this job easy for backup team. Do we have any best practices or recommendation? How are we handling such things in other accounts/engagement? 

So you are talking about GPFS, not JFS2? That changes really a lot (if not everything).

Backing up large GPFS filesystems is not a simple and easy task. I strongly suggest careful planning of backup strategy, taking into account creating snapshots, filesets, splitting backup into smaller tasks and splitting them between multiple GPFS nodes. GPFS tuning might be necessary, too.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

On a Large SAS Analytics server I Managed a long time ago we had 200,000+ files that totaled 24TB , and backup was always a problem . As part of the modernisation I worked with Sas Engineers to split 200,000+ files from one single filesystem , and organised files into multiple filesystems to match business units/function. I then worked with our company backup engineers ( using Netbackup not TSM in this case ) to create multiple netbackup policy's that target specific filesystems to backup . To perform the targeted approach we used the netbackup exclude/include lists feature to ensure consist backups

Prior

/sasprd

/aenv

New Layout

/sasprd/

/aenv/businessunit1

/aenv/businessunit2

/aenv/businessunit3

/aenv/businessunit4

/aenv/businessunit5

/aenv/archive

etc

# Create 5 netbackup policy's businessunit1, businessunit2, etc to target each of children filesystems of /aenva

cat /usr/openv/netbackup/exclude_list.businessunit1

/sas/prd

/aenv/businessunit2

/aenv/businessunit3

/aenv/businessunit4

/aenv/businessunit5

/aenv/archive

cat /usr/openv/netbackup/include_list.businessunit1

/aenv/businessunit1

Additional, as part of the modernisation the project – 1) Configured a dedicated highspeed Network adapter that used only for backup . 2) Setup a netbackup archive policy to prevent the system growing allow engineers to archive data , to netbackup instead of making a backup mv filea filea.notrequired .

I am not a TSM expert, is there way to using include & exclude lists to create targeted policy's in similar manor ?  It would assume that your GPFS Filesystem layout was structured  , not just  one filesystem, and you had way to archive  data offline to long term storage .

Brian Walker 

Thank you everyone for the response. Networker doesn't support GPFS over AIX now in their latest version, so we are taking backup from Linux servers which took 24-25 days. However when network supported those things in the past it took approx 9-10 days with AIX node. Can we do anything at oslevel to make this job easy for backup team. Do we have any best practices or recommendation? How are we handling such things in other accounts/engagement? 

So you are talking about GPFS, not JFS2? That changes really a lot (if not everything).

Backing up large GPFS filesystems is not a simple and easy task. I strongly suggest careful planning of backup strategy, taking into account creating snapshots, filesets, splitting backup into smaller tasks and splitting them between multiple GPFS nodes. GPFS tuning might be necessary, too.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Getting the DEVs to understand that they can't dump a million files into a single directory is 90% of the work.

The other 90% is convincing the service owner that yes, you do need to reorganize that mess, even if it means downtime.

On a Large SAS Analytics server I Managed a long time ago we had 200,000+ files that totaled 24TB , and backup was always a problem . As part of the modernisation I worked with Sas Engineers to split 200,000+ files from one single filesystem , and organised files into multiple filesystems to match business units/function. I then worked with our company backup engineers ( using Netbackup not TSM in this case ) to create multiple netbackup policy's that target specific filesystems to backup . To perform the targeted approach we used the netbackup exclude/include lists feature to ensure consist backups

Prior

/sasprd

/aenv

New Layout

/sasprd/

/aenv/businessunit1

/aenv/businessunit2

/aenv/businessunit3

/aenv/businessunit4

/aenv/businessunit5

/aenv/archive

etc

# Create 5 netbackup policy's businessunit1, businessunit2, etc to target each of children filesystems of /aenva

cat /usr/openv/netbackup/exclude_list.businessunit1

/sas/prd

/aenv/businessunit2

/aenv/businessunit3

/aenv/businessunit4

/aenv/businessunit5

/aenv/archive

cat /usr/openv/netbackup/include_list.businessunit1

/aenv/businessunit1

Additional, as part of the modernisation the project – 1) Configured a dedicated highspeed Network adapter that used only for backup . 2) Setup a netbackup archive policy to prevent the system growing allow engineers to archive data , to netbackup instead of making a backup mv filea filea.notrequired .

I am not a TSM expert, is there way to using include & exclude lists to create targeted policy's in similar manor ?  It would assume that your GPFS Filesystem layout was structured  , not just  one filesystem, and you had way to archive  data offline to long term storage .

Brian Walker 

Thank you everyone for the response. Networker doesn't support GPFS over AIX now in their latest version, so we are taking backup from Linux servers which took 24-25 days. However when network supported those things in the past it took approx 9-10 days with AIX node. Can we do anything at oslevel to make this job easy for backup team. Do we have any best practices or recommendation? How are we handling such things in other accounts/engagement? 

So you are talking about GPFS, not JFS2? That changes really a lot (if not everything).

Backing up large GPFS filesystems is not a simple and easy task. I strongly suggest careful planning of backup strategy, taking into account creating snapshots, filesets, splitting backup into smaller tasks and splitting them between multiple GPFS nodes. GPFS tuning might be necessary, too.

Thank you Russell. 

Size of the directory is approx. 6 GB. Only thing is it having lot of files inside it. Actually it is GPFS file system where back up team is taking backup of around 20TB file system. Incremental backup is taking around 24-25 days to complete and that may be because of those no of inode/files. Do we have solution for such scenario how we can overcome from this situation. This is one of our critical application. You know it is hard to split out the file system. We are trying to find out the solution. We have networker backup tools to backup the data. Do we have any other backup tool/or any other alternate for this problem.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

No one has mentions how fast a network you are running?

Are you running the latest AIX with service packs, GPFS, System firmware?

Have you checked the GPFS and AIX logs?

Can you make a quick network test? to prove you have what you think you have.

Of course, a single threaded network transfer will be much slower that the theoretical network speed.

To get full network speed you MUST go multithreaded - so your backup will have to do the same.

2 million files means 2 million file creation transaction = that could be the killer
- if the backup copy is transitory (i.e. thrown away) then Switching off Logging is a very good idea.

Too many files in a directory is a UNIX problem for decade.
GPFS can scale to very large files systems but the number of files in one directory remains a bottleneck.  I guess SSD disk speed helps.

Next you can:

1. Look for GPFS tuning
2. Engage Lab Services
3. Call AIX and GPFS Support to check the configuration, software levels and look for bottlenecks.

Manoj, you are a known guru, sorry if this is all obvious and you have done this already.

It might help other finding this question

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Thank you Nigel. 

We are almost running everything on latest either n or n-1. We have other GPFS cluster in the environment but those system are not having issue however total file system size is huge comparison to this problematic cluster. Only difference is this particular GPFS file system having millions of files which others GPFS doesn't. 

Even on the same cluster we have two other GPFS file system and they also don't have any issue. But on those two GPFS, file system size is also small comparison to the problematic one.

On same cluster-> Below are the 3 file systems.

Total           Used           Free

  7800.00   6389.03   1410.97  - Good
  2500.00   1740.80    759.20   - Good
 29200.00  24331.22   4868.78   --> Problematic

No one has mentions how fast a network you are running?

Are you running the latest AIX with service packs, GPFS, System firmware?

Have you checked the GPFS and AIX logs?

Can you make a quick network test? to prove you have what you think you have.

Of course, a single threaded network transfer will be much slower that the theoretical network speed.

To get full network speed you MUST go multithreaded - so your backup will have to do the same.

2 million files means 2 million file creation transaction = that could be the killer
- if the backup copy is transitory (i.e. thrown away) then Switching off Logging is a very good idea.

Too many files in a directory is a UNIX problem for decade.
GPFS can scale to very large files systems but the number of files in one directory remains a bottleneck.  I guess SSD disk speed helps.

Next you can:

1. Look for GPFS tuning
2. Engage Lab Services
3. Call AIX and GPFS Support to check the configuration, software levels and look for bottlenecks.

Manoj, you are a known guru, sorry if this is all obvious and you have done this already.

It might help other finding this question

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Thank you Nigel. 

We are almost running everything on latest either n or n-1. We have other GPFS cluster in the environment but those system are not having issue however total file system size is huge comparison to this problematic cluster. Only difference is this particular GPFS file system having millions of files which others GPFS doesn't. 

Even on the same cluster we have two other GPFS file system and they also don't have any issue. But on those two GPFS, file system size is also small comparison to the problematic one.

On same cluster-> Below are the 3 file systems.

Total           Used           Free

  7800.00   6389.03   1410.97  - Good
  2500.00   1740.80    759.20   - Good
 29200.00  24331.22   4868.78   --> Problematic

No one has mentions how fast a network you are running?

Are you running the latest AIX with service packs, GPFS, System firmware?

Have you checked the GPFS and AIX logs?

Can you make a quick network test? to prove you have what you think you have.

Of course, a single threaded network transfer will be much slower that the theoretical network speed.

To get full network speed you MUST go multithreaded - so your backup will have to do the same.

2 million files means 2 million file creation transaction = that could be the killer
- if the backup copy is transitory (i.e. thrown away) then Switching off Logging is a very good idea.

Too many files in a directory is a UNIX problem for decade.
GPFS can scale to very large files systems but the number of files in one directory remains a bottleneck.  I guess SSD disk speed helps.

Next you can:

1. Look for GPFS tuning
2. Engage Lab Services
3. Call AIX and GPFS Support to check the configuration, software levels and look for bottlenecks.

Manoj, you are a known guru, sorry if this is all obvious and you have done this already.

It might help other finding this question

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Hi Manoj

From the details that you have shared, are there two separate issues here?

1) backups of a GPFS file system containing 1.7 million files is slower that backups of similarly sized file systems containing a small number of files

2) backup time increased when backups were moved from an AIX GPFS node to a Linux node. 9-10 days on AIX vs. 24-25 days on Linux.

Is that correct? For problem #2, is the Linux node an NSD client? If so that might explain the degradation assuming that other speeds and feeds are equal between the Linux backup node and the old AIX backup node.

Thank you Nigel. 

We are almost running everything on latest either n or n-1. We have other GPFS cluster in the environment but those system are not having issue however total file system size is huge comparison to this problematic cluster. Only difference is this particular GPFS file system having millions of files which others GPFS doesn't. 

Even on the same cluster we have two other GPFS file system and they also don't have any issue. But on those two GPFS, file system size is also small comparison to the problematic one.

On same cluster-> Below are the 3 file systems.

Total           Used           Free

  7800.00   6389.03   1410.97  - Good
  2500.00   1740.80    759.20   - Good
 29200.00  24331.22   4868.78   --> Problematic

No one has mentions how fast a network you are running?

Are you running the latest AIX with service packs, GPFS, System firmware?

Have you checked the GPFS and AIX logs?

Can you make a quick network test? to prove you have what you think you have.

Of course, a single threaded network transfer will be much slower that the theoretical network speed.

To get full network speed you MUST go multithreaded - so your backup will have to do the same.

2 million files means 2 million file creation transaction = that could be the killer
- if the backup copy is transitory (i.e. thrown away) then Switching off Logging is a very good idea.

Too many files in a directory is a UNIX problem for decade.
GPFS can scale to very large files systems but the number of files in one directory remains a bottleneck.  I guess SSD disk speed helps.

Next you can:

1. Look for GPFS tuning
2. Engage Lab Services
3. Call AIX and GPFS Support to check the configuration, software levels and look for bottlenecks.

Manoj, you are a known guru, sorry if this is all obvious and you have done this already.

It might help other finding this question

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Hi Chris,

Both are the same issue. In first step we have tried locally on the servers itself to copy one directory (Having million files) from one place to another and it was taking lot of time. Even simple command taking time on that directory. 

Second we are taking backup of same file system via a backup tool which also taking time. AIX is on GPFS is now not supported by backup tool so we moved from AIX server to Linux server to take the backup of GPFS file system. Yes it is NSD client mapped via TCPIP.

Hi Manoj

From the details that you have shared, are there two separate issues here?

1) backups of a GPFS file system containing 1.7 million files is slower that backups of similarly sized file systems containing a small number of files

2) backup time increased when backups were moved from an AIX GPFS node to a Linux node. 9-10 days on AIX vs. 24-25 days on Linux.

Is that correct? For problem #2, is the Linux node an NSD client? If so that might explain the degradation assuming that other speeds and feeds are equal between the Linux backup node and the old AIX backup node.

Thank you Nigel. 

We are almost running everything on latest either n or n-1. We have other GPFS cluster in the environment but those system are not having issue however total file system size is huge comparison to this problematic cluster. Only difference is this particular GPFS file system having millions of files which others GPFS doesn't. 

Even on the same cluster we have two other GPFS file system and they also don't have any issue. But on those two GPFS, file system size is also small comparison to the problematic one.

On same cluster-> Below are the 3 file systems.

Total           Used           Free

  7800.00   6389.03   1410.97  - Good
  2500.00   1740.80    759.20   - Good
 29200.00  24331.22   4868.78   --> Problematic

No one has mentions how fast a network you are running?

Are you running the latest AIX with service packs, GPFS, System firmware?

Have you checked the GPFS and AIX logs?

Can you make a quick network test? to prove you have what you think you have.

Of course, a single threaded network transfer will be much slower that the theoretical network speed.

To get full network speed you MUST go multithreaded - so your backup will have to do the same.

2 million files means 2 million file creation transaction = that could be the killer
- if the backup copy is transitory (i.e. thrown away) then Switching off Logging is a very good idea.

Too many files in a directory is a UNIX problem for decade.
GPFS can scale to very large files systems but the number of files in one directory remains a bottleneck.  I guess SSD disk speed helps.

Next you can:

1. Look for GPFS tuning
2. Engage Lab Services
3. Call AIX and GPFS Support to check the configuration, software levels and look for bottlenecks.

Manoj, you are a known guru, sorry if this is all obvious and you have done this already.

It might help other finding this question

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

Hi Manoj

From the details that you have shared, are there two separate issues here?

1) backups of a GPFS file system containing 1.7 million files is slower that backups of similarly sized file systems containing a small number of files

2) backup time increased when backups were moved from an AIX GPFS node to a Linux node. 9-10 days on AIX vs. 24-25 days on Linux.

Is that correct? For problem #2, is the Linux node an NSD client? If so that might explain the degradation assuming that other speeds and feeds are equal between the Linux backup node and the old AIX backup node.

Thank you Nigel. 

We are almost running everything on latest either n or n-1. We have other GPFS cluster in the environment but those system are not having issue however total file system size is huge comparison to this problematic cluster. Only difference is this particular GPFS file system having millions of files which others GPFS doesn't. 

Even on the same cluster we have two other GPFS file system and they also don't have any issue. But on those two GPFS, file system size is also small comparison to the problematic one.

On same cluster-> Below are the 3 file systems.

Total           Used           Free

  7800.00   6389.03   1410.97  - Good
  2500.00   1740.80    759.20   - Good
 29200.00  24331.22   4868.78   --> Problematic

No one has mentions how fast a network you are running?

Are you running the latest AIX with service packs, GPFS, System firmware?

Have you checked the GPFS and AIX logs?

Can you make a quick network test? to prove you have what you think you have.

Of course, a single threaded network transfer will be much slower that the theoretical network speed.

To get full network speed you MUST go multithreaded - so your backup will have to do the same.

2 million files means 2 million file creation transaction = that could be the killer
- if the backup copy is transitory (i.e. thrown away) then Switching off Logging is a very good idea.

Too many files in a directory is a UNIX problem for decade.
GPFS can scale to very large files systems but the number of files in one directory remains a bottleneck.  I guess SSD disk speed helps.

Next you can:

1. Look for GPFS tuning
2. Engage Lab Services
3. Call AIX and GPFS Support to check the configuration, software levels and look for bottlenecks.

Manoj, you are a known guru, sorry if this is all obvious and you have done this already.

It might help other finding this question

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.

GPFS handles large numbers of files quite well (I'm just looking at two filesystems with 9M and 12M files).

There are four things that cause problems:
- Directories with large numbers of files are slow to list (both 'ls' and shell wildcard globing), particularly when you start trashing the stat cache.
- Scanning big directories during backups
- Slow metadat disks
- Slow VMs 

For large filesystems, tunning is not an option, it's a must.

Hi Manoj

From the details that you have shared, are there two separate issues here?

1) backups of a GPFS file system containing 1.7 million files is slower that backups of similarly sized file systems containing a small number of files

2) backup time increased when backups were moved from an AIX GPFS node to a Linux node. 9-10 days on AIX vs. 24-25 days on Linux.

Is that correct? For problem #2, is the Linux node an NSD client? If so that might explain the degradation assuming that other speeds and feeds are equal between the Linux backup node and the old AIX backup node.

Thank you Nigel. 

We are almost running everything on latest either n or n-1. We have other GPFS cluster in the environment but those system are not having issue however total file system size is huge comparison to this problematic cluster. Only difference is this particular GPFS file system having millions of files which others GPFS doesn't. 

Even on the same cluster we have two other GPFS file system and they also don't have any issue. But on those two GPFS, file system size is also small comparison to the problematic one.

On same cluster-> Below are the 3 file systems.

Total           Used           Free

  7800.00   6389.03   1410.97  - Good
  2500.00   1740.80    759.20   - Good
 29200.00  24331.22   4868.78   --> Problematic

No one has mentions how fast a network you are running?

Are you running the latest AIX with service packs, GPFS, System firmware?

Have you checked the GPFS and AIX logs?

Can you make a quick network test? to prove you have what you think you have.

Of course, a single threaded network transfer will be much slower that the theoretical network speed.

To get full network speed you MUST go multithreaded - so your backup will have to do the same.

2 million files means 2 million file creation transaction = that could be the killer
- if the backup copy is transitory (i.e. thrown away) then Switching off Logging is a very good idea.

Too many files in a directory is a UNIX problem for decade.
GPFS can scale to very large files systems but the number of files in one directory remains a bottleneck.  I guess SSD disk speed helps.

Next you can:

1. Look for GPFS tuning
2. Engage Lab Services
3. Call AIX and GPFS Support to check the configuration, software levels and look for bottlenecks.

Manoj, you are a known guru, sorry if this is all obvious and you have done this already.

It might help other finding this question

Hi Team,

We are trying to copy a directory which is having lot of files (1760000). It is taking lot of time to copy from one directory to another on the same server. (Approx 1hr 15 min)

Even commands like ls, du also takes longer time to complete. 

Is this normal behavior? or we can do something at OS to speed up the copy. Any suggestion would be highly appreciated.