Authors: @Ananya Gupta @KEDAR KARMARKAR

Table of Contents

1. Introduction
2. What is FTDC in IBM Storage Scale?
3. FTDC Enhancements in Storage Scale 6.0.1
4. NFS Performance and Memory Statistics Collection via Sysmon
5. Core Dump Collection via gpfs.snap
6. Debug Data Collection on Hung Scenario
7. NFS Client Debug Script
8. Ganesha Trace Utility (ganeshatracectl)
9. Conclusion

1. Introduction

Troubleshooting issues in large-scale NFS environments often requires collecting diagnostic data from multiple system components, including the NFS server, GPFS filesystem, network infrastructure, and client systems. Because problems can originate at different layers of the stack, identifying the root cause can be complex and time-consuming.

In a typical NFS deployment, issues may arise from several areas, such as:

• NFS server processes

• GPFS filesystem operations

• Network communication

• Client workloads

Effective troubleshooting therefore requires collecting a variety of diagnostic data, including:

• Server performance statistics

• Process stack traces

• Crash dumps

• Network traces

• Client-side debugging information

Manually gathering this information across multiple systems can be inefficient, inconsistent, and sometimes incomplete, which may delay root cause analysis.

To improve diagnostics and observability for CES NFS environments, IBM Storage Scale 6.0.1 enhances the First Time Data Capture (FTDC) framework. These enhancements automate the collection of critical diagnostic data, ensuring that relevant information is captured when issues occur.

The enhancements focus on improving:

• Observability of NFS server behavior

• Automated diagnostic data collection

• Troubleshooting efficiency in production environments

2. What is FTDC in IBM Storage Scale?

First Time Data Capture (FTDC) is a diagnostic framework that automatically collects critical system information when issues occur.

Instead of relying on manual data collection after a problem is detected, FTDC captures relevant diagnostic data at the time the issue occurs. This proactive approach ensures that important runtime information is preserved for analysis.

By automatically gathering logs, statistics, and system state information, FTDC significantly reduces the time required for:

• Problem identification
• Root cause analysis
• Support case resolution

3. FTDC Enhancements in Storage Scale 6.0.1

IBM Storage Scale 6.0.1 introduces several enhancements to improve diagnostics and observability for CES NFS (NFS-Ganesha) environments.

4. NFS Performance and Memory Statistics Collection via Sysmon

Overview:

Understanding runtime behavior of the NFS server is critical when diagnosing performance issues or unexpected workload patterns.

IBM Storage Scale 6.0.1 adds support for collecting periodic NFS-Ganesha performance and memory statistics through mmsysmon using the ganesha_stats utility.

These statistics expose key runtime characteristics of NFS server activity, including:

• Protocol operations

• Authentication activity

• Client workload patterns

• Memory usage

This capability improves diagnostics and observability for NFS workloads and provides valuable context when investigating performance issues or unexpected server behavior in production environments.

To minimize overhead, The functionality is disabled by default, and can be enabled when additional diagnostic data collection is required.

Configuration:

NFS performance statistics collection is controlled through configuration parameters in:

/ts/appadmin/mmsysmonitor/config/mmsysmonitor.conf

under the [nfs] section.

These options can be configured either directly in the mmsysmonitor.conf file or through the mmchconfig command.

Enabling or Disabling Statistics Collection:

The NFS statistics collection options can be enabled or disabled using the mmchconfig command.

Example – enable memory statistics collection:

mmchconfig mmhealth-nfs-collectnfsmemperf=yes --force

Example – disable memory statistics collection:

mmchconfig mmhealth-nfs-collectnfsmemperf=no --force

Mapping between configuration options and mmchconfig parameters:

Example commands:

Enable general NFS performance statistics:

mmchconfig mmhealth-nfs-collectnfsperf=yes --force

Enable per-client statistics:

mmchconfig mmhealth-nfs-collectnfsclientperf=yes --force

Disable per-client statistics:

mmchconfig mmhealth-nfs-collectnfsclientperf=no --force

These configuration changes control how mmsysmon collects NFS diagnostic statistics.

General NFS Performance Statistics:

When collectnfsperf = true, the system periodically collects core NFS-Ganesha statistics using:

ganesha_stats inode
ganesha_stats v3_full
ganesha_stats v4_full
ganesha_stats auth
ganesha_stats fsal gpfs

These statistics provide information about inode cache behavior, protocol activity, authentication requests, and GPFS FSAL performance metrics.

Client Performance Statistics:

When collectnfsclientperf = true, additional statistics are collected for each connected client:

ganesha_stats list_clients
ganesha_stats client_all_ops <client_ip>

These metrics help identify clients generating high request volumes or unusual workload patterns.

Memory Statistics:

When collectnfsmemperf = true, memory usage statistics are collected using:

ganesha_stats mem_stats enable
ganesha_stats mem_stats show

These metrics provide visibility into memory consumption across NFS-Ganesha components and can help identify memory growth or potential memory leaks.

When to Enable Each Option:

Log Location:

/var/adm/ras/nfs_ganesha_perf_stats_<hostname>.log

The log contains periodic snapshots of NFS-Ganesha statistics, providing a historical view of server activity that can assist during problem determination.

5. Core Dump Collection via gpfs.snap

Overview:

When NFS-Ganesha crashes, collecting the core dump and related diagnostic logs is essential for identifying the root cause of the failure. Without these logs, post-mortem debugging of the crash becomes significantly more difficult.

IBM Storage Scale 6.0.1 enhances gpfs.snap to automatically collect NFS-Ganesha core dumps when snap data is gathered after a crash.

This ensures that critical crash logs are captured and included in the gpfs.snap bundle without requiring additional manual steps.

In addition to the core dump, dependent libraries used by the NFS-Ganesha process are also collected to support core dump analysis.

This enhancement is supported on RHEL 8, RHEL 9, and RHEL 10 environments.

Core Dump Collection:

The mechanism used to collect NFS-Ganesha core dumps depends on the operating system version.

RHEL 8:

For RHEL 8 systems, crash reports are retrieved using the ABRT framework.

The following command is used to list crash reports:

abrt-cli list --detailed

Recent crash directories are collected from:

/var/spool/abrt/

Only directories that meet the following conditions are collected:

• Contain Ganesha-related crashes
• Generated within the last five days

RHEL 9 and RHEL 10:

For RHEL 9 and RHEL 10, the systemd coredump service is used.

The following command is used to list available core dumps:

coredumpctl list

The snap process collects:

• The last three core dumps
• Generated within the previous five days

Collection of Dependent Libraries:

When a Ganesha core dump is detected, gpfs.snap also collects the Ganesha executable and its dependent libraries required for core dump analysis.

This ensures that the debugging environment contains the same binaries and libraries that were used when the crash occurred.

The dependency collection is performed using the following script:

/usr/lpp/mmfs/samples/nfs/collect_deps_libs.py

The script collects:

• The Ganesha executable
• All dependent shared libraries

These files are packaged into an archive that can be used for offline debugging of the core dump.

Optional Configuration:

By default, gpfs.snap collects NFS-Ganesha core dumps automatically when available.

Core dump collection can be skipped using the following option:

--skip-nfs-coredump

6. Debug Data Collection on Hung Scenario

Overview:

In certain situations, the NFS-Ganesha process may become unresponsive while the service itself remains running. In such cases, client requests may stop progressing even though the process has not crashed.

Diagnosing these scenarios can be challenging because the issue may involve interactions between the NFS server, the GPFS filesystem, or client workloads.

IBM Storage Scale 6.0.1 enhances diagnostic data collection for these situations by allowing additional debug information to be gathered when a hung condition is detected by mmsysmon.

This capability is controlled through the configuration option 'collectnfscoreonhang'.

Depending on the configuration, the system can collect either lightweight process diagnostics or a full Ganesha core dump together with a GPFS internal dump.

Configuration:

The behavior is controlled through the following configuration parameter in:

/ts/appadmin/mmsysmonitor/config/mmsysmonitor.conf

under the [nfs] section.

This option can also be configured using the mmchconfig command.

Enable core dump collection on hang:

mmchconfig mmhealth-nfs-collectnfscoreonhang=yes --force

Disable core dump collection on hang:

mmchconfig mmhealth-nfs-collectnfscoreonhang=no --force

Diagnostic Collection Behavior:

Diagnostics Collected (Default Mode):

Network state:

netstat -an

NFS-Ganesha statistics:

ganesha_stats inode
ganesha_stats fast
ganesha_stats v3_full
ganesha_stats v4_full
ganesha_stats auth
ganesha_stats fsal gpfs

Cluster-wide waiter information:

mmlsnode -N waiters -L

Recent GPFS statistics:

mmdiag --stats

Process diagnostics:

/proc/<pid>/limits
/proc/<pid>/stack
/proc/<pid>/task/*/stack

These stack traces help identify threads that may be blocked or waiting on resources.

Diagnostics Collected When Core Dump Is Enabled:

When collectnfscoreonhang = true, additional diagnostics are collected.

Ganesha core dump using gcore command

GPFS internal dump using mmdump command.

These logs provide detailed runtime information and can be used for in-depth debugging of the hung NFS-Ganesha process.

7. NFS Client Debug Script

Overview:

Troubleshooting NFS client issues often requires collecting multiple types of diagnostic data, including system statistics, network traces, RPC debug logs, and filesystem information.

Manually gathering this data can be time-consuming and may result in incomplete diagnostics.

IBM Storage Scale 6.0.1 provides a scenario-based NFS client debug script that automates the collection of client-side diagnostic data for common NFS issues.

The script collects relevant client diagnostics based on the selected troubleshooting scenario, helping simplify the process of gathering information required for problem analysis.

Script Availability:

The NFS client debug script is provided as part of the IBM Storage Scale installation package and is located on Storage Scale nodes at:

/usr/lpp/mmfs/samples/nfs/nfs_client_debug_script.py

The script is not installed on NFS client systems by default. When troubleshooting client-side issues, the script should be copied from a Storage Scale protocol node to the affected NFS client system and executed there.

When to Use the Script:

The script should be used when an NFS client experiences issues accessing an NFS export, such as:

• Mount failures
• Unresponsive mount points
• Permission or access errors
• Performance degradation

Command Usage:

The script supports the following command format:

# nfs_client_debug_script.py --scenario {trace_mount_error,trace_mount_hang,access_denied_error,trace_perf_issue}

[--server_ip SERVER_IP] [--nfs_share_path NFS_SHARE_PATH] [--local_mount_path LOCAL_MOUNT_PATH] [--log_directory LOG_DIRECTORY] [--nfs_version NFS_VERSION] [--username USERNAME] [--test_directory TEST_DIRECTORY] [--debug]

Supported Troubleshooting Scenarios:

Scenario Argument Requirements:

Example Commands:

Trace Mount Error:

/usr/lpp/mmfs/samples/nfs/nfs_client_debug_script.py --scenario trace_mount_error --server_ip 192.168.0.141 --nfs_share_path /mnt/gpfs0/nfs2 --local_mount_path /mnt/test2 --nfs_version 4

Trace Mount Hang:

/usr/lpp/mmfs/samples/nfs/nfs_client_debug_script.py --scenario trace_mount_hang --server_ip 192.168.0.141 --local_mount_path /mnt/test2

Trace Access Denied Error:

/usr/lpp/mmfs/samples/nfs/nfs_client_debug_script.py --scenario access_denied_error --server_ip 192.168.0.141 --local_mount_path /mnt/test2

Trace Performance Issue:

/usr/lpp/mmfs/samples/nfs/nfs_client_debug_script.py --scenario trace_perf_issue --server_ip 192.168.0.141 --local_mount_path /mnt/test2

System Requirements:

• Python 3.6 or later
• tcpdump
• nfsstat, vmstat, df utilities
• rpcdebug utility
• Root privileges
• Sufficient free disk space in the log directory

8. Ganesha Trace Utility (ganeshatracectl)

Overview:

Troubleshooting complex NFS issues may require collecting detailed runtime information from the NFS-Ganesha process and the surrounding system environment.

This information can include process state, network activity, filesystem statistics, and GPFS diagnostics.

IBM Storage Scale provides the ganeshatracectl trace utility to simplify the collection and management of NFS-Ganesha trace logs and diagnostic data.

The utility consolidates multiple tracing and monitoring capabilities into a single tool, allowing comprehensive diagnostic data to be collected during troubleshooting scenarios.

Utility Location:

The trace utility is provided as part of the IBM Storage Scale installation package and is available on Storage Scale protocol nodes at:

/usr/lpp/mmfs/samples/nfs/ganeshatracectl.py

The utility configuration file is located at:

/usr/lpp/mmfs/samples/nfs/ganeshatracectl.conf

Command Usage:

The utility can be executed using the following syntax:

/usr/lpp/mmfs/samples/nfs/ganeshatracectl.py <command>

Supported Commands:

Utility Capabilities:

The ganeshatracectl utility provides several diagnostic capabilities, including:

• Trace collection of NFS-Ganesha process state
• Network traffic capture using tcpdump
• Performance monitoring of memory usage and system statistics
• Error monitoring from NFS-Ganesha logs
• RPC debugging and NFS statistics collection
• GPFS diagnostic data collection, including filesystem statistics and VFS usage
• System health checks, including network and RPC service validation

The utility also supports log rotation to manage diagnostic data generated during extended tracing sessions.

Typical Workflow:

A typical troubleshooting workflow using ganeshatracectl involves configuring the trace parameters, starting trace collection, reproducing the issue, and stopping the trace to gather the diagnostic logs.

1. Edit the configuration file and review the trace parameters in:

   /usr/lpp/mmfs/samples/nfs/ganeshatracectl.conf

2. Start trace collection:

   /usr/lpp/mmfs/samples/nfs/ganeshatracectl.py start

   • When tracing starts, the utility creates a timestamped trace directory and begins collecting diagnostic data.
   • Logs are stored at: /var/log/ganesha_trace/ganesha_trace_<hostname>_<timestamp>
   • Example output: Trace collection started!
3. Reproduce the NFS issue being investigated.

   Allow the script to run until the NFS issue is reproduced.

4. Stop trace collection:

   /usr/lpp/mmfs/samples/nfs/ganeshatracectl.py stop

   Stopping the trace performs the following actions:

   • Stops all running trace operations
   • Packages the collected logs into a compressed archive
   • Optionally removes the temporary trace directory

   Example output:

   Stopping tracing... This may take up to 10 minutes.
   Creating a tar.gz archive of log files.
   Archive created successfully: /var/log/ganesha_trace/ganesha_trace_<hostname>_<timestamp>.tar.gz

Trace Output Location:

When tracing starts, the script creates a timestamped directory under:

/var/log/ganesha_trace/

Example: /var/log/ganesha_trace/ganesha_trace_<hostname>_<timestamp>

All diagnostic logs generated during tracing are stored in this directory.

Collected Log Files:

The trace directory contains multiple log files capturing different aspects of the system and the NFS-Ganesha process.

Example files generated:

The main trace activity can be monitored in real time by observing:

ganeshatracectl_script.log

Trace Archive:

When tracing is stopped, the collected logs are packaged into a compressed archive:

/var/log/ganesha_trace/ganesha_trace_<hostname>_<timestamp>.tar.gz

This archive contains the complete set of collected diagnostic logs and can be shared for further debugging or analysis.

System Requirements:

The ganeshatracectl script requires the following utilities to be available on the system:

• Python 3
• lsof
• gdb
• strace
• tcpdump
• These utilities are used by the script to collect process information, stack traces, and network diagnostics.

9. Conclusion

The FTDC enhancements in IBM Storage Scale 6.0.1 significantly improve diagnostics and observability for CES NFS environments. By automating the collection of critical diagnostic data, administrators and support teams can reduce troubleshooting time and accelerate root cause analysis in production deployments.