Viewing and Examining Logs in IBM Spectrum Fusion
Introduction
The Log Collector is responsible for gathering and delivering a set of IBM Spectrum Fusion resources. Most resources are JSON strings returned from the Kubernetes REST API calls. Pod logs, and hardware logs can also be collected. Logs can be requested from the IBM Spectrum Fusion UI or from the command line.
Architecture
Log Collector is designed to collect information on any resource in Fusion. The basic steps to retrieve a log is
- POST a JSON request to https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector
- The request includes the group/version/kind of a resource
- The results of the POST request is a job id (ie 61227a1a-c27c-4b40-afd9-ce1165e30474)
- GET the status of that jobid https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector/61227a1a-c27c-4b40-afd9-ce1165e30474
- Once the status is finished, GET the results as a .zip file https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector/61227a1a-c27c-4b40-afd9-ce1165e30474/results -o archive.zip
Collection sets stay on the server for 24 hours and are then automatically deleted.
Example
# find the route to use for collecting log from the command line
> oc get route logcollector
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
logcollector logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com
# Submit a request for information on the logcollector deployment
> curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\
{
"requests": {
"lc-dep": {
"type": "k8s-resource",
"description": "Gather data about the logcollector deployment",
"group": "apps",
"version": "v1",
"kind": "deployments",
"namespace": "ibm-spectrum-fusion-ns",
"name": "logcollector"
}
}
}
====>
{"jobid":"61227a1a-c27c-4b40-afd9-ce1165e30474"}
# Check the current status of the collection
> curl -k --request GET https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector/61227a1a-c27c-4b40-afd9-ce1165e30474
====>
{
"requests": {
"lc-dep": {
"description": "Gather data about the logcollector deployment",
"group": "apps",
"kind": "deployments",
"name": "logcollector",
"namespace": "ibm-spectrum-fusion-ns",
"type": "k8s-resource",
"version": "v1"
}
},
"status": "finished",
"completed": {
"lc-dep": {
"description": "Gather data about the logcollector deployment",
"group": "apps",
"kind": "deployments",
"name": "logcollector",
"namespace": "ibm-spectrum-fusion-ns",
"type": "k8s-resource",
"version": "v1"
}
},
"inprogress": {},
"created": "2021-09-12T13:18:52.960006356Z",
"updated": "2021-09-12T13:18:55.421193368Z",
"errors": {},
"expires": "2021-09-13T13:18:55.421193452Z",
"size": 5617966,
"freespace": "74147086336 / 128283815936, (57.8%)"
}
# Download the results of the collection request
> curl -k https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector/61227a1a-c27c-4b40-afd9-ce1165e30474/res
ults -o archive.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5486k 100 5486k 0 0 408k 0 0:00:13 0:00:13 --:--:-- 456k
===============================================================================
# Files included in the collection, includes the deployment, replicaset, pod, and pod logs
> unzip -l archive.zip
Archive: archive.zip
Length Date Time Name
--------- ---------- ----- ----
154 09-12-2021 09:18 lc-dep/deployments/events.logcollector.json
6403 09-12-2021 09:18 lc-dep/deployments/logcollector.json
154 09-12-2021 09:18 lc-dep/deployments/replicasets/events.logcollector-6446f68445.json
5649 09-12-2021 09:18 lc-dep/deployments/replicasets/logcollector-6446f68445.json
48213666 09-12-2021 09:18 lc-dep/deployments/replicasets/pods/containers.logcollector-6446f68445-gs775/logcollector.log
212 09-12-2021 09:18 lc-dep/deployments/replicasets/pods/containers.logcollector-6446f68445-gs775/logcollector.previous.log
154 09-12-2021 09:18 lc-dep/deployments/replicasets/pods/events.logcollector-6446f68445-gs775.json
8268 09-12-2021 09:18 lc-dep/deployments/replicasets/pods/logcollector-6446f68445-gs775.json
783 09-12-2021 09:18 status.json
--------- -------
48235443 9 files
Implied Requests
Many requests will return information on the requested resource and the resources it owns
- Deployments, replicasets, daemonsets, & daemons will include the dependent resources down to the current and previous pod logs (as shown in the previous example)
- Requesting group/version/kind/namespace without a name will get all instances of that kind in that namespace
# Submit a request for information on all deployments in the ibm-spectrum-fusion-ns namespace
> curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\
{
"requests": {
"all-deps": {
"type": "k8s-resource",
"description": "Gather data about all deployments in ibm-spectrum-fusion-ns",
"group": "apps",
"version": "v1",
"kind": "deployments",
"namespace": "ibm-spectrum-fusion-ns"
}
}
}
- Requesting group/version/namespace/namespacename will get all major resources in that namespace. Currently major resources are
- Deployments
- Daemonsets
- Daemons
- Stateful sets
- Config maps
- Services
- Routes
- Persistent volume claims
# Submit a request for information on all major resources in the ibm-spectrum-fusion-ns namespace
> curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\
{
"requests": {
"all-res": {
"type": "k8s-resource",
"description": "Gather data about all major resources in ibm-spectrum-fusion-ns",
"group": "apps",
"version": "v1",
"namespace": "ibm-spectrum-fusion-ns"
}
}
}
Other Types of Logs
Network switches
# Submit a request for all network switch traces
> curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\
{
"requests": {
"network-switches": {
"type": "switch-logs",
"description": "All network switch logs",
"name": "all"
}
}
}
IMM reports
- OneCli misc ffdc
- OneCli inventory getinfor
# Submit a request for all IMM traces
> curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\
{
"requests": {
"imm-logs": {
"type": "imm-logs",
"description": "All IMM logs",
"name": "all"
}
}
}
OCP Audit logs
- openshift-apiserver
- kube-apiserver
- oauth-apiserver
# Submit a request for all audit logs
> curl -k --request POST 'https://logcollector-ibm-spectrum-fusion-ns.apps.cps-r81-9-46-123-89.rtp.raleigh.ibm.com/api/v1/logcollector' --header 'Content-Type: application/json' --data '\
{
"requests": {
"audit-logs": {
"type": "audit-logs",
"description": "All audit logs",
"nodes": "all"
}
}
}
Methods of Collection
Logs can be collected in 4 basic ways, from easiest to hardest.
- Automatically - When a Call Home ticket is generated, and customers have automatic log collection enabled, there should be a collection of logs that match the specific problem area associated with the ticket in ECURep.
- UI - The customer can also obtain log collection zip files by going to a UI page and clicking one of six possible collection sets. The resulting zip file can be manually uploaded to ECURep.
- Command line Log Collector curl request - If the existing collection sets do not contain the information you need, you can code a curl request for logs as shown above, with the resulting zip file uploaded to ECURep
- General CLI commands - If the Log Collector does not have the ability to collect the information you need, then you can give a set of CLI commands to the customer, ask them to login as kubeadmin and run the commands, collect the results in a file, and then upload the file to ECURep. There are also log collection capabilities on the Spectrum Scale UI.
Collection Contents
All automatic and UI collection sets will contain
- oc get nodes
- kickstart config map
- rackinfo config map
- persistent volumes
The size of this data on a test system is 13k zipped.
The Administration UI-only collection set (450M size) contains
- openshift-logging namespace dump
- OCP audit logs
The Backup and Restore UI and automatic collection set (30M) contains
- ibm-spectrum-protect-plus-ns namespace dumps
- baas and spp-velero namespace dumps
- sppmanagers, controlplanebackup, and backupmanager custom resources
The Compute UI and automatic collection set (30M) contains
- ibm-spectrum-fusion-ns namespace dump
If you are facing a more general problem (like UI) and need to request a collection set, this is a generic one that will contain all the resources for Fusion.
The Network UI and automatic collection set (30M) contains
- ibm-spectrum-fusion-ns namespace dump
- vlans, links, switches custom resources
The Storage UI and automatic collection set (70M) contains
- ibm-spectrum-scale, ibm-spectrum-scale-csi, ibm-spectrum-scale-operator, openshift-logging, openshift-monitoring, and ibm-spectrum-fusion-ns namespace dumps
The System Health UI-only collection set (300M) contains
- ibm-spectrum-fusion-ns namespace dump
- node metrics (oc adm top nodes)
- pod metrics (oc adm top pods)
- baremetalhosts, machinesets, machines, and nodenetworkconfigurationpolicies custom resources
- IMM logs
- all switch logs
Note: This collection set can take up to 3 hours to be collected
Config Maps
Most aspects of log collection are controlled by Config Maps defined in the ibm-spectrum-fusion-ns namespace:
- isf-serviceability-operator-allow-tickets - the list of CRITICAL events that will open tickets, as well as default logs to be collected. If a particular event is opening too many tickets, this config map can be updated to turn off all ticket creation by that event.
- isf-serviceability-operator-collection-sets - definition of which logs belong to which collection set
- isf-serviceability-operator-components - list of components recognized by Serviceability
- isf-serviceability-operator-logs - list of pre-defined logs for convenience
Sample Download
UI
The Logs UI page can be reached from the Serviceability menu in the upper left corner of the IBM Spectrum Fusion console.
There are currently 6 collection sets.
To create a collection set, select one or more sets, and then click Create selected log sets.
The collection job will start
The page will be updated when the collection job is finished
To download the logs, choose the download icon
The file is downloaded as a zip file with the job id of the request