Hyperconverged Infrastructure (HCI)

Viewing and Examining Logs in IBM Spectrum Fusion

By Byron Williams posted Thu October 28, 2021 01:46 PM

  

Viewing and Examining Logs in IBM Spectrum Fusion

Introduction

The Log Collector is responsible for gathering and delivering a set of IBM Spectrum Fusion resources. Most resources are JSON strings returned from the Kubernetes REST API calls. Pod logs, and hardware logs can also be collected. Logs can be requested from the IBM Spectrum Fusion UI or from the command line.

Architecture

Log Collector is designed to collect information on any resource in Fusion. The basic steps to retrieve a log is

  1. POST a JSON request to https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector
  2. The request includes the group/version/kind of a resource
  3. The results of the POST request is a job id (ie 61227a1a-c27c-4b40-afd9-ce1165e30474)
  4. GET the status of that jobid https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector/61227a1a-c27c-4b40-afd9-ce1165e30474
  5. Once the status is finished, GET the results as a .zip file https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector/61227a1a-c27c-4b40-afd9-ce1165e30474/results -o archive.zip

Collection sets stay on the server for 24 hours and are then automatically deleted.

Example

# find the route to use for collecting log from the command line > oc get route logcollector NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD logcollector logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com # Submit a request for information on the logcollector deployment > curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\ { "requests": { "lc-dep": { "type": "k8s-resource", "description": "Gather data about the logcollector deployment", "group": "apps", "version": "v1", "kind": "deployments", "namespace": "ibm-spectrum-fusion-ns", "name": "logcollector" } } } ====> {"jobid":"61227a1a-c27c-4b40-afd9-ce1165e30474"} # Check the current status of the collection > curl -k --request GET https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector/61227a1a-c27c-4b40-afd9-ce1165e30474 ====> { "requests": { "lc-dep": { "description": "Gather data about the logcollector deployment", "group": "apps", "kind": "deployments", "name": "logcollector", "namespace": "ibm-spectrum-fusion-ns", "type": "k8s-resource", "version": "v1" } }, "status": "finished", "completed": { "lc-dep": { "description": "Gather data about the logcollector deployment", "group": "apps", "kind": "deployments", "name": "logcollector", "namespace": "ibm-spectrum-fusion-ns", "type": "k8s-resource", "version": "v1" } }, "inprogress": {}, "created": "2021-09-12T13:18:52.960006356Z", "updated": "2021-09-12T13:18:55.421193368Z", "errors": {}, "expires": "2021-09-13T13:18:55.421193452Z", "size": 5617966, "freespace": "74147086336 / 128283815936, (57.8%)" } # Download the results of the collection request > curl -k https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector/61227a1a-c27c-4b40-afd9-ce1165e30474/res ults -o archive.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5486k 100 5486k 0 0 408k 0 0:00:13 0:00:13 --:--:-- 456k =============================================================================== # Files included in the collection, includes the deployment, replicaset, pod, and pod logs > unzip -l archive.zip Archive: archive.zip Length Date Time Name --------- ---------- ----- ---- 154 09-12-2021 09:18 lc-dep/deployments/events.logcollector.json 6403 09-12-2021 09:18 lc-dep/deployments/logcollector.json 154 09-12-2021 09:18 lc-dep/deployments/replicasets/events.logcollector-6446f68445.json 5649 09-12-2021 09:18 lc-dep/deployments/replicasets/logcollector-6446f68445.json 48213666 09-12-2021 09:18 lc-dep/deployments/replicasets/pods/containers.logcollector-6446f68445-gs775/logcollector.log 212 09-12-2021 09:18 lc-dep/deployments/replicasets/pods/containers.logcollector-6446f68445-gs775/logcollector.previous.log 154 09-12-2021 09:18 lc-dep/deployments/replicasets/pods/events.logcollector-6446f68445-gs775.json 8268 09-12-2021 09:18 lc-dep/deployments/replicasets/pods/logcollector-6446f68445-gs775.json 783 09-12-2021 09:18 status.json --------- ------- 48235443 9 files

Implied Requests

Many requests will return information on the requested resource and the resources it owns

    • Deployments, replicasets, daemonsets, & daemons will include the dependent resources down to the current and previous pod logs (as shown in the previous example)
    • Requesting group/version/kind/namespace without a name will get all instances of that kind in that namespace
# Submit a request for information on all deployments in the ibm-spectrum-fusion-ns namespace > curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\ { "requests": { "all-deps": { "type": "k8s-resource", "description": "Gather data about all deployments in ibm-spectrum-fusion-ns", "group": "apps", "version": "v1", "kind": "deployments", "namespace": "ibm-spectrum-fusion-ns" } } }
  • Requesting group/version/namespace/namespacename will get all major resources in that namespace. Currently major resources are
    • Deployments
    • Daemonsets
    • Daemons
    • Stateful sets
    • Config maps
    • Services
    • Routes
    • Persistent volume claims
# Submit a request for information on all major resources in the ibm-spectrum-fusion-ns namespace > curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\ { "requests": { "all-res": { "type": "k8s-resource", "description": "Gather data about all major resources in ibm-spectrum-fusion-ns", "group": "apps", "version": "v1", "namespace": "ibm-spectrum-fusion-ns" } } }

Other Types of Logs

Network switches

# Submit a request for all network switch traces > curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\ { "requests": { "network-switches": { "type": "switch-logs", "description": "All network switch logs", "name": "all" } } }


IMM reports

    • OneCli misc ffdc
    • OneCli inventory getinfor
# Submit a request for all IMM traces > curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\ { "requests": { "imm-logs": { "type": "imm-logs", "description": "All IMM logs", "name": "all" } } }


OCP Audit logs

  • openshift-apiserver
  • kube-apiserver
  • oauth-apiserver
# Submit a request for all audit logs > curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\ { "requests": { "audit-logs": { "type": "audit-logs", "description": "All audit logs", "nodes": "all" } } }

Methods of Collection

Logs can be collected in 4 basic ways, from easiest to hardest.

  1. Automatically - When a Call Home ticket is generated, and customers have automatic log collection enabled, there should be a collection of logs that match the specific problem area associated with the ticket in ECURep.
  2. UI - The customer can also obtain log collection zip files by going to a UI page and clicking one of six possible collection sets. The resulting zip file can be manually uploaded to ECURep.
  3. Command line Log Collector curl request - If the existing collection sets do not contain the information you need, you can code a curl request for logs as shown above, with the resulting zip file uploaded to ECURep
  4. General CLI commands - If the Log Collector does not have the ability to collect the information you need, then you can give a set of CLI commands to the customer, ask them to login as kubeadmin and run the commands, collect the results in a file, and then upload the file to ECURep. There are also log collection capabilities on the Spectrum Scale UI.

Collection Contents

All automatic and UI collection sets will contain

  • oc get nodes
  • kickstart config map
  • rackinfo config map
  • persistent volumes

The size of this data on a test system is 13k zipped.

The Administration UI-only collection set (450M size) contains

  • openshift-logging namespace dump
  • OCP audit logs

The Backup and Restore UI and automatic collection set (30M) contains

  • ibm-spectrum-protect-plus-ns namespace dumps
  • baas and spp-velero namespace dumps
  • sppmanagers, controlplanebackup, and backupmanager custom resources

The Compute UI and automatic collection set (30M) contains

  • ibm-spectrum-fusion-ns namespace dump

If you are facing a more general problem (like UI) and need to request a collection set, this is a generic one that will contain all the resources for Fusion.

The Network UI and automatic collection set (30M) contains

  • ibm-spectrum-fusion-ns namespace dump
  • vlans, links, switches custom resources

The Storage UI and automatic collection set (70M) contains

  • ibm-spectrum-scale, ibm-spectrum-scale-csi, ibm-spectrum-scale-operator, openshift-logging, openshift-monitoring, and ibm-spectrum-fusion-ns namespace dumps

The System Health UI-only collection set (300M) contains

  • ibm-spectrum-fusion-ns namespace dump
  • node metrics (oc adm top nodes)
  • pod metrics (oc adm top pods)
  • baremetalhosts, machinesets, machines, and nodenetworkconfigurationpolicies custom resources
  • IMM logs
  • all switch logs

Note: This collection set can take up to 3 hours to be collected

Config Maps

Most aspects of log collection are controlled by Config Maps defined in the ibm-spectrum-fusion-ns namespace:

  • isf-serviceability-operator-allow-tickets - the list of CRITICAL events that will open tickets, as well as default logs to be collected. If a particular event is opening too many tickets, this config map can be updated to turn off all ticket creation by that event.
  • isf-serviceability-operator-collection-sets - definition of which logs belong to which collection set
  • isf-serviceability-operator-components - list of components recognized by Serviceability
  • isf-serviceability-operator-logs - list of pre-defined logs for convenience

Sample Download

Sample Archive
Sample Archive

Sample Archive 3


UI


The Logs UI page can be reached from the Serviceability menu in the upper left corner of the IBM Spectrum Fusion console.

Fusion Menu


There are currently 6 collection sets.

Collection Sets


To create a collection set, select one or more sets, and then click Create selected log sets.

Creating a collection set


The collection job will start

Collection Job


The page will be updated when the collection job is finished

Collection finished


To download the logs, choose the download icon

Download Logs


The file is downloaded as a zip file with the job id of the request

Downloaded
0 comments
19 views

Permalink