IBM Cloud Pak for AIOps (AIOps) comes with two types of Netcool Connector to integrate with your existing Netcool deployment:
- IBM Netcool Operations Insight ObjectServer - for connecting to Netcool/OMNIbus 8.1 ObjectServers
- IBM Tivoli Netcool/Impact - for connecting to Netcool/Impact 7.1 clusters
This blog provides some tips for setting up these two essential Connectors and how to get up and running quickly and successfully.
IBM Netcool Operations Insight ObjectServer Connector
The IBM Netcool Operations Insight ObjectServer Connector (ObjectServer Connector) provides a robust and efficient way to relay events from Netcool/OMNIbus to AIOps. This section outlines some key pointers on how to successfully set up an ObjectServer Connector instance.
PREPARING YOUR OBJECTSERVER
When using the ObjectServer Connector with an ObjectServer, you need to prepare your ObjectServer prior to connection to ensure the two-way synchronisation of events works correctly. Create a file on the Netcool server with the following content:
alter table alerts.status add column AIOpsAlertId varchar(64);
go
alter table alerts.status add column AIOpsState varchar(16);
go
create or replace trigger group NetcoolConnectorAutomation;
go
alter trigger group NetcoolConnectorAutomation set enabled true;
go
CREATE OR REPLACE TRIGGER netcool_connector_closed_alerts
GROUP NetcoolConnectorAutomation
PRIORITY 15
COMMENT 'Netcool Connector handle closed alerts in AIOps'
AFTER UPDATE ON alerts.status
FOR EACH ROW
WHEN get_prop_value('ActingPrimary') %= 'TRUE' and
new.AIOpsState = 'closed'
begin
delete from alerts.status via new.Identifier;
end;
go
alter table alerts.status add column ScopeID varchar(255);
go
Next, import the SQL file into your ObjectServer:
$OMNIHOME/bin/nco_sql -server AGG_P -username root -password netcool < aiops_prep.sql
NOTE: You may see some errors if fields already exist. Such errors can be safely ignored.
ENSURING A SUCCESSFUL CONNECTION
The next part of the ObjectServer Connector setup involves adding the user credentials and the target system host name and port. If the target Netcool host name is not resolveable by DNS, then it is likely your connection will fail. This sub-section outlines why this can happen and provides a solution on how to ensure a successful connection.
The ObjectServer has a primary port (default 4100) and a secondary port, called the IDUC port. All connecting clients connect to the ObjectServer's primary port and so-called IDUC clients make a secondary connection to the ObjectServer's IDUC port. IDUC stands for Inserts, Deletes, Updates, and Control. The IDUC port is used to publish details of any inserts, deletes, updates, and control information that occur within an ObjectServer's defined granularity period (60 seconds by default). Netcool/Gateways and the Netcool/OMNIbus Native Event List (thick client) are the main users of the IDUC port.
The ObjectServer Connector contains a Netcool/Gateway for Message Bus as part of its componentry. When a Netcool Gateway connects to an ObjectServer, the following outlines the sequence that is followed to set up the connection:
- The Gateway connects to host name and port specified - eg.
netcool-server on port 4100
- The ObjectServer negotiates the connection and provides its IDUC channel host name and port - eg.
netcool-server on port 69153 (ephemeral)
- The Gateway makes a second connection to the ObjectServer's IDUC port on the host name and port provided by the ObjectServer
Because of this second connection made by the Gateway to the IDUC channel, the Gateway needs to be able to resolve the host name provided by the ObjectServer to AIOps, to complete the connection protocols and succeed in its connection. If the Gateway cannot resolve the host name given by the ObjectServer, the connection will fail. Even if you specify the connection to the ObjectServer using its IP address, while the initial connection may succeed, the secondary connection to the IDUC port will fail, as the Gateway will attempt to connect to the host name given by the ObjectServer.
NOTE: if there is a firewall between AIOps and Netcool/OMNIbus, it is also necessary to fix the IDUC port via the Iduc.ListeningPort property in the ObjectServer's properties file. If unset, the ObjectServer will randomly select an ephemeral port and hence the port for the secondary connection is unknown ahead of ObjectServer start-up and subject to change if the ObjectServer subsequently restarts.
Issues with host name resolution can be an issue commonly experienced in test and demonstration environments where the virtual machine running the Netcool components has been freshly provisioned and has not been added to DNS. One solution is to simply add the Netcool host to the DNS server used by AIOps.
Another solution is to add an entry to the AIOps pods' hosts files so that the host name given can be resolved locally within the pod. The way you do this is different if you're running AIOps on OpenShift or Linux.
For the OpenShift-based version of AIOps, you will need to modify the Deployment YAML post-creation. For the Linux-based version of AIOps, you will need to add a ConfigMap to your AIOps system containing the host name and IP address of the target Netcool VM if running AIOps on Linux. The steps for both are included below.
ADDING TO HOSTS FILE: AIOPS ON OPENSHIFT
If you are running AIOps on OpenShift, you will create the Connector instance first, then modify the deployment YAML that subsequently gets created. After you have completed the below steps to create the instance, use the following steps to get connectivity working.
- Log in to the OpenShift console
- Go to Workloads → Deployments
- Search for your Connector Deployment
- The ObjectServer Connector Deployment will have a name like:
netcool-conn-634cf3c1-1fde-4d5d-8ac2-4790335802f8
- The Impact Connector Deployment will have a name like:
ibm-grpc-impact-connector-806b595c-7b69-4a7d-8a5b-73875857a4bc
- Click on your Connector Deployment and select the YAML tab
- Add the following lines (example) to your
spec.template.spec section
hostAliases:
- ip: 10.38.200.150
hostnames:
- netcool-server.ibm.com
- netcool-server
TIP: if you search for the following line in the YAML, you can paste the hostAliases entry in on the line just above it:
serviceAccount: connector-87fd5502-127e-4acc-b22b-df4ce66847a2
IMPORTANT: the indentation in the hostAliases entry is critical. Ensure that you copy and paste it accurately.
After saving your updated Deployment YAML, navigate to Workloads → Pods and search for: netcool-conn
You will see the Connector pod has restarted. If you click into it and click on the Terminal tab, you can inspect the local hosts file and see your newly added entry:
$ cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 locahost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.128.3.101 netcool-conn-ef17fd0a-72cd-42b8-90ac-8d46f1495c3d-6565687dgkdv8
# Entries added by HostAliases.
10.38.200.150 netcool-server.ibm.com netcool-server
sh-4.4$
Your ObjectServer Connector should now successfully be able to connect to your Netcool/OMNIbus 8.1 ObjectServer.
ADDING TO HOSTS FILE: AIOPS ON LINUX
If you are running AIOps on Linux, you will need to create the following ConfigMap and restart the coredns deployment using the commands below:
kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns-custom
namespace: kube-system
data:
default.server: |
netcool-server.ibm.com {
hosts {
10.38.200.150 netcool-server.ibm.com
fallthrough
}
}
netcool-server {
hosts {
10.38.200.150 netcool-server
fallthrough
}
}
EOF
kubectl -n kube-system rollout restart deployment coredns
NOTE: Keep a copy of the above in case you need to modify or add to your DNS settings. Each time you apply the above, it will replace the existing configuration.
After creating and applying this ConfigMap, if you subsequently log into the pod and try to curl to the Netcool server, you will be able to see that the pod is now able to resolve the host name to the correct IP address:
# kubectl exec -n aiops netcool-conn-598d9b27-fba2-43fe-9a12-38c825181140-7975c974njgll -it -- /bin/sh
sh-4.4$ curl -v -k https://netcool-server.ibm.com
* Rebuilt URL to: https://netcool-server.ibm.com/
* Trying 10.38.200.150...
* TCP_NODELAY set
...
Your ObjectServer Connector should now successfully be able to connect to your Netcool/OMNIbus 8.1 ObjectServer.
DEPLOYMENT OPTIONS
You should opt to run the Connector instance locally, within AIOps. The circumstances in which you'd run the Connector remotely are few, and typically involve connectivity and the placement of components with respect to firewalls and security constraints. Generally speaking however, it's much simpler and easier to maintain a system where all the Connectors are deployed locally, hence this is the preferred approach.
SETTING UP THE MAPPING
The final step in setting up your ObjectServer Connector instance is to configure the mapping. The following is an example mapping that can be used for ObjectServer Connector instances:
(
$isIPAddr := function($i){ $contains($i,/^[0-9]+.[0-9]+.[0-9]+.[0-9]+$/)};
{
"summary": alert.@Summary,
"deduplicationKey": alert.@Identifier,
"sender": {
"service": alert.@Agent,
"name": alert.@Manager
},
"resource": {
"name": alert.@Node = "" ? alert.@NodeAlias = "" ? undefined : alert.@NodeAlias : alert.@Node,
"location": alert.@Location = "" ? undefined : alert.@Location,
"ipAddress": $isIPAddr(alert.@NodeAlias) ? alert.@NodeAlias : $isIPAddr(alert.@Node) ? alert.@Node : undefined,
"hostname": $not($isIPAddr(alert.@Node)) ? alert.@Node : undefined,
"sourceId": alert.@NodeAlias = "" ? undefined : alert.@NodeAlias,
"service": alert.@Service = "" ? undefined : alert.@Service,
"port": alert.@PhysicalPort = 0 ? undefined : alert.@PhysicalPort,
"physicalslot": alert.@PhysicalSlot = 0 ? undefined : alert.@PhysicalSlot,
"physicalcard": alert.@PhysicalCard = "" ? undefined : alert.@PhysicalCard,
"scopeId": alert.@ScopeID = "" ? undefined : alert.@ScopeID
},
"type": {
"eventType": alert.@Type = 2 ? "resolution" : alert.@Type = 4 ? "resolution" : "problem",
"classification": alert.@EventId = "" ? alert.@AlertGroup: alert.@EventId
},
"eventCount": alert.@Tally,
"signature": alert.@Identifier,
"firstOccurrenceTime": alert.@FirstOccurrence,
"lastOccurrenceTime": alert.@LastOccurrence,
"severity": alert.@Type = 13 ? 2 : alert.@Severity < 0 ? 1 : alert.@Severity = 0 ? 2 : alert.@Severity < 6 ? alert.@Severity + 1 : alert.@Severity >= 6 ? 6,
"state": alert.@Severity = 0 ? "clear" : "open",
"acknowledged": alert.@Acknowledged = 1 ? true : false,
"expirySeconds": alert.@ExpireTime,
"suppressed": alert.@SuppressEscl = 4 ? true : false,
"customer": alert.@Customer,
"details": {
"alertGroup": alert.@AlertGroup,
"alertKey": alert.@AlertKey,
"locationName": alert.@LocationName = "" ? undefined : alert.@LocationName,
"locationId": alert.@LocationID <= 0 ? "0" : $string(alert.@LocationID)
},
"insights": [
{
"details": {
"lastProcessedEventOccurrenceTime": alert.@LastOccurrence
},
"id": "event-occurrence",
"type": "aiops.ibm.com/insight-type/deduplication-details"
}
]
}
)
NOTES:
- If you set the
ScopeID field in OMNIbus, you can map this to the resource.scopeId attribute in AIOps and AIOps will automatically perform scope-based grouping based on this field. Note that this doesn't preclude the event participating in other scope-based correlations within AIOps but it provides a way to leverage scope-based grouping work already done in Netcool/OMNIbus.
- Certain attributes like
severity in AIOps are mandatory. If an event arrives into AIOps with an undefined severity, the event will be dropped. The above mapping logic ensures it always ends up with a value.
- It is prudent to map
ExpireTime to expirySeconds and SuppressEscl to suppressed AIOps so that these attributes can be leveraged in both places. Note that AIOps has a built-in automation that automatically clears alerts expirySeconds after the last occurrence timestamp of the alert. The suppressed attribute is also a default attribute in AIOps and is used to indicate if an alert is suppressed or not.
- Custom fields should be mapped as sub-attributes under the
details attribute, per the above example. Note that at the time of writing (version 4.6.1), only string variables are accepted by details sub-attributes. If you want to map a Netcool/OMNIbus integer field to AIOps therefore, then you must cast the result to a string per the above example. Other types will be accepted by the ObjectServer Connector in a future release.
IBM Tivoli Netcool/Impact Connector
The IBM Tivoli Netcool/Impact Connector (Impact Connector) is used to connect AIOps to a Netcool/Impact 7.1 cluster so that you can create Netcool/Impact policies in AIOps to send events for processing by Netcool/Impact. This is extremely handy for implementing custom automation, work-flows, or event enrichment from external sources of data.
This section outlines some tips to ensure you get up and running quickly with the Impact Connector.
SPECIFY THE ENDPOINTS
The first part of the configuration is to add the URLs for the GUI server and back-end server. Note that the format expected is:
https://<hostname>:<port>
NOTE: You do not append /ibm/console to the end of the URL.
The first URL required is that of the GUI server. The default port for the GUI server is port 16311, hence an example would be:
https://netcool-server.ibm.com:16311
It is important to use the fully qualified domain name of the host running Netcool/Impact and this FQDN must match that on the certificate entered in a later step. If your system cannot resolve the FQDN of the Netcool/Impact host because it's not in DNS, then you will need to update the pod's local hosts file. The instructions to do this are provided in the previous section.
The second URL required is that of the Netcool/Impact back-end API. The default port used is 9081, hence an example would be:
https://netcool-server.ibm.com:9081
ENTER THE IMPACT SERVER USERNAME AND PASSWORD
Next, you need to add the login details for your Impact user. The default username impactadmin and the password will be whatever you set it to when you installed Netcool/Impact.
ADD SELF-SIGNED CERTIFICATES
The next step can be omitted if the Impact server is using certificates issued by a proper issuing authority. Demo and test environments however are typically using self-signed certificates. In this case, you need to paste in the certificates into the Connector configuration field to validate the connection ("Impact certificates (self-signed certificates only)"). If you omit this step, the connection will not succeed.
To get the content for this field, log in to the VM running Netcool/Impact and use the output of the following two commands concatenated together:
echo '' | openssl s_client -connect `hostname`:16311 -showcerts 2>&1 | sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p'
echo '' | openssl s_client -connect `hostname`:9081 -showcerts 2>&1 | sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p'
Finally, you should opt to run the Connector instance locally, within AIOps. The circumstances in which you'd run the Connector remotely are few, and typically involve connectivity and the placement of components with respect to firewalls and security constraints. Generally speaking however, it's much simpler and easier to maintain a system where all the Connectors are deployed locally, hence this is the preferred approach.
CONFIGURE THE AIOPS REST DSA
When AIOps makes a successful connection to Netcool/Impact during the Connector instantiation process, it creates a number of assets within Netcool/Impact. These assets include a number of Impact policies that make it simpler to handle event payloads coming from AIOps and for writing back to AIOps. These Impact policies can be found in the Policies section of Netcool/Impact and are all prefixed with the name: "AIOPS_".
Another asset created by AIOps in Netcool/Impact is a Data Source Adaptor (DSA) of type "RESTful API". The DSA name is created in the format of AIOPs-ConnectorName-ConnectorID. It will have a name something like "AIOps-impact-806b595c-7b69-4a7d-8a5b-73875857a4bc" and will be found in the Data Model tab of Netcool/Impact.
Although it is initially created, its configuration needs to be finished off manually. The following steps outline how to do this.
- Create an API Key in AIOps by first logging in to AIOps as an admin user - eg. cpadmin
- Click on your profile photo in the top-right corner and select "Profile and settings".
- Click on "API key" in the top-right corner.
- Generate a new API key and save the value to a safe place. This API key will be used by the Impact DSA to access AIOps.
- Authenticate to your AIOps platform system - either your OpenShift cluster or your Linux based platform:
OpenShift-based AIOps:
- Log in to your OpenShift console as user kubeadmin (or as an administrator) and select "kube:admin" ⇒ "Copy login command".
- Click on "Display Token" from the new tab that opens and copy the "oc login" command from the section: "Log in with this token".
- Paste the "oc login" command into a command line window to authenticate with your OpenShift cluster.
Linux-based AIOps:
- SSH to your primary control plane node in your AIOps cluster. Once authenticated, you will be able to run oc commands from here.
- Use your API key created earlier to run the following command to define an environment variable: ZENAPIKEY
ZENAPIKEY=$(echo "cpadmin:ZWzOk1ddkIzJmgHKayjO6zGJ4Y1Spik2zcbIltaA" | base64 -w 0)
- Using the FQDN of your AIOps system (CPD_ROUTE), run the following to test your ZENAPIKEY and echo it to the command line:
CPD_ROUTE=cpd-aiops.apps.aiops-4.cp.fyre.ibm.com
TENANT_ID=cfd95b7e-3bc7-4006-a4a8-a73a79c71255
curl -kv -H "Authorization: ZenApiKey ${ZENAPIKEY}" -H "x-tenant-id: ${TENANT_ID}" "https://${CPD_ROUTE}/aiops/api/issue-resolution/v1/alerts/"
echo ${ZENAPIKEY}
- If you are successful, you should see a spurious output to the screen of XML formatted alerts.
- Note how the above curl command uses a header called "Authorization".
- The authorisation token Netcool/Impact will use will be in the same format - example:
ZenApiKey Y3BhZG1pbjpaV3pPazFkZGtJekptZ0hLYXlqTzZ6R0o0WTFTcGlrMlBD1234567890==
- Save your generated ZenApiKey above to a safe place. It will be used again in a later step.
- Edit the AIOps DSA and update the "Source" section:
- Edit the "Authorization" header and update the "Header Value" with your ZenApiKey you created earlier. Note that you need the string "ZenApiKey", followed by a space character, followed by the computed ZENAPIKEY value - example:
ZenApiKey Y3BhZG1pbjpaV3pPazFkZGtJekptZ0hLYXlqTzZ6R0o0WTFTcGlrMlBD1234567890==
- Click on the Save button to save your changes.
- Right-click on the DSA name in the list on the left and select: "Test Connection".
- If you've done everything correctly, you should see a "Connection OK" message:
--
Congratulations; you are now ready to start working with Netcool/Impact and AIOps!
Watch the video of this blog on YouTube: https://youtu.be/eQtG37j1Iow
--
ACKNOWLEDGEMENTS: thanks to Matt Thornhill for providing the example ConfigMap.