File and Object Storage

 View Only

Protocol Problem Determination Guide for IBM Spectrum Scale™ - Log Collection

By Archive User posted Wed July 04, 2018 01:40 PM



In this article we will see how to Collect logs for analysing any problem that is seen on Spectrum Scale with respect to Authentication and FILE protocols. So in short we will see issues likely to be seen in the areas:
1. Authentication
2. SMB Access
3. NFS Access
4. Data Ownership/Access problems

Data Collection

To diagnose any problem it is necessary to gather relevant information from the cluster. Collection of debugging information, such as configuration files and logs, can be achieved by using the gpfs.snap command. This command gathers data about GPFS, operating system information, and information for each of the protocols. It also collects AUTHENTICATION related data like Authentication configuration and logs. To collect only authentication traces use the following command,
# gpfs.snap -–protocol authentication

Authentication Data captured by gpfs.snap command

The following authentication data is always obtained by the gpfs.snap command:

1. The output of these commands:

  • # mmuserauth service list

  • # mmuserauth service check --data-access-method all --nodes cesNodes

  • # mmuserauth service check --data-access-method all --nodes cesNodes --server-reachability

  • # systemctl status ypbind

  • # systemctl status sssd

  • # ps aux | grep keystone

  • # lsof -i

  • # sestatus

  • # systemctl status firewalld

  • # systemstl status iptables

  • # /usr/lpp/mmfs/bin/net ads info

  • 2. The following files:

  • /etc/nsswitch.conf

  • /etc/ypbind.conf

  • /etc/idmapd.conf

  • /etc/sssd/*

  • /etc/krb5.conf

  • /etc/krb5.keytab

  • /etc/firewalld/*

  • /var/log/sssd/*

  • Collecting Logs for LDAP or NIS Based Authentication.

    This will collect configuration files for the SSSD Component

  • /etc/sssd/sssd.conf

  • /etc/krb5.conf (if LDAP Kerberos authentication configured)

  • /etc/krb5.keytab (if LDAP Kerberos authentication configured)

  • Log files are:

  • /var/log/sssd/sssd.log

  • /var/log/sssd/sssd_nss.log

  • /var/log/sssd/sssd_LDAPDOMAIN.log (if LDAP authentication configured)

  • /var/log/sssd/sssd_NISDOMAIN.log (if NIS authentication configured)

  • Note: For more information on SSSD log files, see Red Hat Linux documentation

    Winbind (AD based authentication schemes)

    Configuration Files are:

  • /etc/krb5.conf

  • /etc/krb5.keytab (if AD with kerberized NFS is configured)

  • Log files are:

  • /var/adm/ras/log.wb-

  • /var/adm/ras/log.winbindd-dcconnect

  • /var/adm/ras/log.winbindd-idmap

  • /var/adm/ras/log.winbindd

  • Authentication configuration failures

    Pre configuration

  • Pre-requisites are not meet and hence CLI fails
  • Environment related failures:
    -- Network related
    -- Administrative credentials requirements

  • Post configuration

  • Verify if the validation command outputs,

  • # mmuserauth service check --data-access-method all --nodes cesNodes
    # mmuserauth service check --data-access-method all --nodes cesNodes --server-reachability

  • Verify user can authenticate over the SMB protocol.

  • Access Failures

    To help debug Access failures we must collect traces. These are Logs at high levels.
    The command, #mmprotocoltrace is used for collecting traces.
    Ideally we need to Start collecting trace, recreate the issue which we want to debug and then Immediately stop collecting trace.
    We can collect traces for debugging problems related to SMB, Winbind, Network and Object tracing.
    NFS tracing is achieved by increasing the log level, repeating the issue, capturing the log file, and then restoring the log level.
    After the issue is recreated by running the gpfs.snap command either with no arguments or with the --protocol nfs argument, the NFS logs are captured.

    Data ownership display problems

  • Check data ownership on protocol server

  • Check data ownership on client mounting the export

  • Validate the user and its group resolves to the same UIDNumber and GIDNumber on both – protocol server and client.

  • Check if appropriate group memberships are returned

  • Based on the variant of protocol access ( eg: NFSv3 Vs NFSv4 ) ensure the necessary pre-requisite setup is done on protocol server and client

  • #Softwaredefinedstorage