IT Automation

 View Only

Splunk Enterprise Integration with Cloud Pak for Watson AIOps

By SREEDHAR KODALI posted Tue January 11, 2022 01:15 PM

  
Co author: Eric Erpenbach

Introduction

IBM Cloud Pak for Watson AIOps is a scalable Ops platform that deploys advanced, explainable AI across an IT Operations toolchain. It can reduce operational costs significantly by proactively assessing, diagnosing and resolving incidents emanating from infrastructure and operations management before they become serious problems. It enables Site Reliability Engineers (SREs) and CIOs to uncover hidden insights from multiple sources of data – like logs, metrics and events and delivers them directly into team collaboration tools like Slack, Microsoft Teams, and ServiceNow in near real-time.

In version 3.2, IBM Cloud Pak for Watson AIOps provides a new AI analytics engine that delivers rich detection of issues with reduced noise. It also provides an easy training oversight, along with the ability to manage all training data. It introduces statistical baselines for log anomaly detection while improving the overall usability and scalability.

Data connections enable IBM Cloud Pak for Watson AIOps to collect log, metric and event data from various incoming sources and communicate how day-to-day IT operations are performing against a baseline model established by AI model training. Watson AIOps recognizes a number of input sources for data integration such as ELK, Humio, LogDNA, Splunk, etc.

Splunk Integration

A Splunk input source can provide log and event data for Watson AIOps. Log data is used for detecting anomalies, whereas event data is used for processing events and alerts.

Splunk Enterprise enables searching, analyzing and visualizing data gathered from an IT infrastructure. It indexes data from servers, applications, databases, network devices, and virtual machines that are part of an IT infrastructure. It can collect data from different infrastructure end points whether they reside locally, remotely or on cloud. Upon data ingestion from log files and other sources, the received data is parsed and indexed so queries can be made against the indexed data.

Splunk Enterprise can be deployed either as a single instance or multiple instances. A single instance deployment is useful for testing and evaluation purposes where multiple instances deployment is necessary for production needs. In a single instance deployment, all aspects of processing data from input through indexing to search is handled by only one instance.

Splunk Enterprise employs forwarders to get data from remote machines. Splunk Universal Forwarder is a forwarder that requires minimal resources and resides on the machine where data originates. It consumes remote data and sends it to Splunk Enterprise for indexing.

This blog post covers in detail the steps to integrate a single instance Splunk Enterprise deployment with IBM Cloud Pak for Watson AIOps for sourcing log data for anomaly detection.

Integration Scenario

Figure1 below captures a simple integration scenario involving Splunk Enterprise 8.2.3 running on a standalone Red Hat 8.4 Linux VM and IBM Cloud Pak for Watson AIOps 3.2 deployed on Red Hat OpenShift 4.7 cluster. A Splunk Universal Forwarder 8.2.3 deployed on a second Red Hat 8.4 Linux VM forwards system logs to Splunk Enterprise instance running on the first VM for indexing.


Let’s quickly see various steps to deploy each one of these integration entities and connect between them.

Splunk Standalone Instance

Follow the instructions laid out in Splunk Enterprise official documentation to install version 8.2.3 on the first Red Hat 8.4 Linux VM.

To secure the Splunk platform, turn on HTTPS encryption for Splunk Web by creating a /opt/splunk/etc/system/local/web.conf file by setting enableSplunkWebSSL option to true and then restart Splunk Enterprise instance:


Splunk Web interface (Figure 2) is available @ https://<splunk-standalone-hostname>:8000 with default self-signed certificate:

Enable token authentication by logging to the Splunk Web with admin credentials and then create a new token for AIOps integration:

Select settings > tokens > enable token authentication > click New Token > (Figure 3) fill-in the create token form fields appropriately, most notably the user, description and expiration > click create button > copy the text from the token field


To start monitoring localhost /var/log directory for events, select settings > data inputs > files & directories > click new local file & directory button > enter /var/log in the text box (Figure 4) and then review and submit the form:



For receiving data from remote VMs into the Splunk Enterprise for indexing and querying, go to settings > forwarding and receiving and then click on +Add new (Figure 5) link against Configure receiving under Receive data section:

Enter the default port of 9997 to receive data from forwarders and click save button.

Splunk Forwarder

Next, login to the second Red Hat Linux m/c with admin credentials and install the matching version of Splunk Universal Forwarder by following instructions laid out in the official documentation.

To configure forwarder to send data to the Splunk Enterprise standalone instance hosted on the first Linux VM, run splunk command with add forward-server option by supplying hostname:port of the standalone hostname:


Next define inputs to the forwarder by invoking splunk command with add monitor option and supplying system log directory such as /var/log:


Check host attribute field in the search dashboard of Splunk Web to verify whether newly added forwarder host is listed.

AIOps Analytics

Follow the instructions outlined in IBM Cloud Pak for Watson AIOps 3.2 knowledge center to install Starter Profile of AI Manager on Red Hat OpenShift version 4.6 or above.

After AI Manager install is successfully done, login to the OpenShift cluster using oc login command on the boot node:


To login to the AI Manager console, you need to retrieve admin password and URL to access the AI Manager console:


Enter the URL in a browser to access the AI Manager console (Figure 6) and login with admin username and password retrieved above:

Splunk Connection

After logging into the AI Manager console, click Data and tool connections from either the left hamburger menu under Define group or tile available on the main console page. Click Add connection and select Logs category and choose Splunk add connection entry:

This pops up a brief Splunk configuration section overview on the right side. Click connect button to proceed. In the Add connection details page, fill-in various connection entries (Figure 8) appropriately with Splunk setup details gathered previously:

> Name: Connection name used for display; for ex, Splunk-Enterprise-Test-Connection
> Description: (optional) Describe the nature of this connection in few; for ex, Test Splunk Enterprise integration with Watson AIOps
> Splunk service URL: URL for accessing REST API of the previously setup Splunk Enterprise standalone instance; for ex, https://<splunk-standalone-hostname>:8089
   Note: 8089 is the API service port for Splunk Enterprise server
> Authentication type: You can select either User ID/password or Token; preferably choose Token type
> Token: Token value to access Splunk REST API; enter token retrieved during the Splunk standalone instance setup
> Certificate: (optional) Certificate to verify SSL/TLS connection to Splunk
   Note: Retrieve SSL certificate from the Splunk Enterprise server using openssl command


> Filters: (optional) Custom search query in Search Processing Language (SPL) to run on Splunk index; leave this field blank
> Base parallelism: Number of search requests that can be made in parallel; leave this field to default value of 1
   Note: Set base parallelism to something below maximum number of concurrent searches possible with the above Splunk Enterprise setup.   Refer to /opt/splunk/etc/system/default/limits.conf for further details.
> Sampling rate: The frequency at which data is sampled from Splunk in seconds for processing live data; leave this field to default value of 60 seconds.

Click Next to proceed to optional Field mapping section (Figure 9) – you can leave this field to default for this simple integration scenario

Click Test connection button to validate the field values entered. You should see Test succeeded message with a tick mark.

Click Next to proceed to AI training and change risk scoring data section (Figure 10). Leave Data flow off for now. You can set mode to Historical data for initial AI training and turn on Data flow when you are ready to undertake model training.


Click Done to complete the connection setup.

Conclusion

In this blog post, we have covered simple integration scenario involving Splunk Enterprise server, Splunk Universal forwarder and Cloud Pak for Watson AIOps and associated steps to test the same. This would allow you to quickly setup Splunk for monitoring your IT infrastructure and Watson AIOps for processing system logs in near real-time and providing actionable insights.

References

For further details, refer to the following links:

(1) Install IBM Cloud Pak for Watson AIOps Starter Profile – https://www.ibm.com/docs/en/cloud-paks/cloud-pak-watson-aiops/3.2.0?topic=manager-starter-installation

(2) Creating Splunk Connections – https://www.ibm.com/docs/en/cloud-paks/cloud-pak-watson-aiops/3.2.0?topic=connections-splunk

(3) Install Splunk Enterprise – https://docs.splunk.com/Documentation/Splunk/8.2.3/SearchTutorial/InstallSplunk

(4) Install Splunk Universal Forwarder – https://docs.splunk.com/Documentation/Forwarder/8.2.3.1/Forwarder/Installanixuniversalforwarder








0 comments
58 views

Permalink