Hyperconverged Infrastructure (HCI)

Configuring and debugging Callhome in IBM Spectrum Fusion

By Subrahmanya Chari Paradani posted Wed October 27, 2021 02:05 AM

  
How to configure and debug Callhome in IBM Spectrum Fusion


IBM Spectrum Fusion is a hyperconverged appliance that helps you to work with its intuitive management graphical user interface for networking resources that associated with the appliance.

This tutorial focuses on how you can configure and debug Callhome function in a IBM Spectrum Fusion appliance.

You can  also refer to the IBM Documentation for IBM Spectrum Fusion.

So, let us get started as follows.

1. Basic Functions
A. Create a Salesforce ticket for critical events.
The Event manager will post a request to callhomeclient to create a ticket from:
Based on the critical events that are received and the configuration in the allow_tickets.yaml file, the sample payload for the callhome structure is as follows:

type LogDetails struct {
Location string `json:"location,omitempty"`
JobID string `json:"jobid,omitempty"`
Status string `json:"status,omitempty"`
Requests map[string]map[string]string `json:"requests,omitempty"`
}

type Description struct {
CallHomeType string `json:"callHomeType"`
ErrorCode string `json:"errorCode"`
ErrorType string `json:"errorType"`
HostName string `json:"hostName"`
Severity string `json:"severity"`
ErrorText string `json:"errorText"`
Fault string `json:"fault"`
}
type CallhomeEventPayload struct {
EventID string `json:"event_id"`
EventTime string `json:"event_time"`
EventTimeMs int64 `json:"event_time_ms"`
EventType string `json:"event_type"`
LocalEventTime string `json:"local_event_time"`
Component string `json:"component"`
State string `json:"state"`
Fault string `json:"fault"`
SerialNum string `json:"serialnum"`
ModelType string `json:"modeltype"`
LogDetails LogDetails `json:"logdetails"`
Description Description `json:"description"`
EventData string `json:"eventdata"`
}
Based on the payload data that is received by callhomeclient from the eventmanger, callhomeclient internally converts this data to the format required for callhome_connect POST request and sends to callhome_connect server from the “https://esupport.ibm.com/connect/api/v1” request.

B. Logs collection
Collects the logs required for debugging the problem that is reported from logcollector and upload the diagnostics data to ecurep for service engineers.

C. Create test ticket
Provides test capability to create a test ticket, which ensures the entitlement of rack from the following post URL
2. Example of Callhomepayload
An example of callhomepayload is as follows:
{
"EventID":"9c3c75cc-fc8a-4a23-980f-4f964fe81965 EventTime":"2021-09-08T13":"44":"29Z EventTimeMs":"1631108669906 EventType":"service LocalEventTime":"2021-09-08T13":"44":"29Z Component":"isf-compute-operator-controller-manager State":"hardware failure Fault":"non_recoverable SerialNum":"racki23 ModelType":"9155-F01 LogDetails":{
"Location":"JobID":"Status":"Requests":"map"[
"loglel00":"map"[
"list-entry":"must-gather source-list":"isf-collection-sets type":"list-entry"
]"loglel01":"map"[
"list-entry":"compute source-list":"isf-collection-sets type":"list-entry"
]
]
}"Description":{
"CallHomeType":"TEST ErrorCode":"ErrorType":"criticalerror HostName":"isf-compute-operator-controller-manager-8446cf7fb8-66cf7 Severity":"1 ErrorText":"BMYCO0013 - (fd8c.215d.178e.c0de.3a68.ddff.fe4f.72e5)":"getting monitoring data failed",
"error":"unable to create ssh connection to imm":"ssh":"handshake failed":"EOF at 2021-09-08T13":"44":"29Z Fault":
}"EventData":labels(isf_uid=9c3c75cc-fc8a-4a23-980f-4f964fe81965,
alertname=BMYCO0013,
identifier=fd8c.215d.178e.c0de.3a68.ddff.fe4f.72e5,
"severity=CRITICAL",
controllerinstance=isf-compute-operator-controller-manager-8446cf7fb8-66cf7,
"controllername=isf-compute-operator-controller-manager",
isf_node=compute-13.isf-racki.rtp.raleigh.ibm.com),
annotations(isf_ibm_mtm=9155-F01,
isf_ibm_serialnum=racki23,
"isf_ticket_id=pending",
"loglel00=isf-collection-sets":"must-gather",
"message=BMYCO0013 - (fd8c.215d.178e.c0de.3a68.ddff.fe4f.72e5)":"getting monitoring data failed",
"error":"unable to create ssh connection to imm":"ssh":"handshake failed":"EOF at 2021-09-08T13":"44":29Z,
labels.isf.ibm.com/isf_node=compute-13.isf-racki.rtp.raleigh.ibm.com,
"isf_fixed=true",
isf_oem_mtm=7X02CTO1WW,
"isf_ticket_requested_at=2021-09-08T13":"44":29Z,
"isf_hw_address=fd8c":"215d":"178e":"c0de":"3a68":"ddff":"fe4f":72e5,
"loglel01=isf-collection-sets":"compute",
isf_oem_serialnum=J101DACW)
}

3. Callhome page
● UI console configuration
Click Serviceability -> Callhome. See the following screen capture.

Click Set up call home on the Call home page. The following screen is displayed.


On the Service level tab, you will see these options:

        • Full Service: Creates a ticket in Salesforce and collects the problem data and uploads to ecurep server.
        • Open Support tickets only: Creates a ticket in Salesforce without collecting any troubleshooting information.
Select the option you want and click Next.
Select the checkbox to agree to IBM’s privacy policy.

On the Company contact tab, provide the relevant values or information in the fields on this tab. See the following screen capture for reference.


Click Next to go to the next tab.

On the System information tab, provide the relevant values or information in the fields. See the following screen capture for reference.




After you provide the relevant values in all the tabs, click Submit to continue.
The Call home page will show all the values that you provided earlier. See the following screen capture for reference.


If you want to turn off the Call Home feature, click Turn off call home. The Turn off call home window will prompt you to confirm your action.

Note: If you want to also remove the configuration information, select the Remove configuration information checkbox.

Click Turn off. See the following screen capture for reference.

4. Events UI
When the ticket is created, the Events page will be populated with the Salesforce ticket number. If call home is turned off by the customer and if any critical events is raised later then the Support ticket column will display the status against the event as “callhome disabled”. See the following screen capture for reference.


Typically, a Salesforce ticket will be displayed as shown in the following screen capture:

For example, a ticket subject may contain the subject as follows:
“From: (77P000B - 9155-S01): Testing full service log collection with a capital S”

Here, “77P000B” is the serial number of the device and “9155-S01” is the model number.

If subject does not contain the serial number and the model number, then the tickets are created from the rack. See the following screen capture for reference.
5. Typical issues seen in callhomeclient

1: Entitlement issues:

When a rack or a device is not entitled correctly, the log will contain errors as follows:

Log:
2021/09/01 15:25:42.174 [1;31m[E][0m [collectlogs.go:240] Error text [Failing Machine Vendor: [IBM], Type: [9155], Serial: [79H0CFG] is NOT REGISTERED in the [RIKER_REG] system. Detail(s): [Riker's Trasaction Id: Z30k8ZuAyjT0HhWh0acf91OlfqD7e7hM, Riker's Message: nested exception is: javax.ejb.EJBException: See nested exception; nested exception is: java.lang.IllegalArgumentException: No account info was found from asset search].]

2021/09/01 15:25:42.174 [1;44m[D][0m [callhome.go:354] error in creating ticket [created] Client.Authentication.NotAuthorized:Failing Machine Vendor: [IBM], Type: [9155], Serial: [79H0CFG] is NOT REGISTERED in the [RIKER_REG] system. Detail(s): [Riker's Trasaction Id: Z30k8ZuAyjT0HhWh0acf91OlfqD7e7hM, Riker's Message: nested exception is: javax.ejb.EJBException: See nested exception; nested exception is: java.lang.IllegalArgumentException: No account info was found from asset search].

2: Unable to create a ticket in Salesforce:

To create a ticket in Salesforce, the callhomeconnect REST call has three states:
a) Send required to create ticket called “service” request
b) Wait for ticket creation on Salesforce and pool every 30 seconds called “last_contact” request
c) Confirm the Salesforce ticket creation “confirm_response" request
Log:
2021/09/15 07:05:42.311 [1;44m[D][0m [collectlogs.go:553] CollectLogs(): error from ConfirmUnconfirmedEvents: could not find eventid[service_asset_event_IBM-ISF5639SPS00_78H0CFG_e4c8e412-f1a5-4663-bae6-34a0d728f9a3] in last_contact response
2021/09/15 07:05:42.311 [1;44m[D][0m [collectlogs.go:557] Sleeping 30 seconds before next query of PMR state
2021/09/15 07:06:12.311 [1;44m[D][0m [collectlogs.go:562] Salesforce ID was not returned in 10 minutes.
2021/09/15 07:06:12.311 [1;44m[D][0m [callhome.go:354] error in creating ticket [created] Salesforce ID was not returned in 10 minutes

If ticket creation failed, then logs will contain “error in creating ticket [created] Salesforce ID was not returned in 10 minutes”

3: Log Collection didn’t complete on time:


Sometimes, logs collection will not complete on time or may fail. In such cases, logs will be seen as follows:
Log:
2021/09/09 12:55:49.868 [1;44m[D][0m [collectlogs.go:620] Sleeping 5 sec before querying log collection state
2021/09/09 12:55:54.880 [1;44m[D][0m [collectlogs.go:650] response: {"requests":{"loglel00":{"list-entry":"must-gather","source-list":"isf-collection-sets","type":"list-entry"},"loglel01":{"list-entry":"compute","source-list":"isf-collection-sets","type":"list-entry"},"logswl00":{"name":"all","type":"switch-logs"}},"status":"running","completed":{},"inprogress":{"loglel00":{"list-entry":"must-gather","source-list":"isf-collection-sets","type":"list-entry"},"loglel01":{"list-entry":"compute","source-list":"isf-collection-sets","type":"list-entry"},"logswl00":{"name":"all","type":"switch-logs"}},"created":"2021-09-09T12:55:49.865839547Z","updated":"2021-09-09T12:55:49.865839547Z","errors":{},"expires":"0001-01-01T00:00:00Z","size":-1,"freespace":"17676894208 / 26843545600, (65.9%)"}
2021/09/09 12:55:54.919 [1;44m[D][0m [collectlogs.go:663] status: running
2021/09/09 12:55:54.933 [1;31m[E][0m [collectlogs.go:728] Log Collection 37702394-9400-4127-a790-ab5571099af4 not finished

4: ISF-UID for event not found:

Sometimes, the ISF-UID will not be present in the Events list. In such cases, the status of callhome client will not update correctly in the UI. Therefore, the logs will be as follows:
Log:
2021/09/09 12:55:54.937 [1;44m[D][0m [callhome.go:354] error in creating ticket [TS003765019] Event not found


Hope that you are now able to configure and debug the Callhome function in IBM Spectrum Fusion. If you have any questions about this article, please reach out to me at 

0 comments
10 views

Permalink