Community
Search Options
Search Options
Log In
Skip to main content (Press Enter).
Sign in
Skip auxiliary navigation (Press Enter).
IT Automation
Topic groups
AIOps
Expertise Connect(EC) Group
Global IT Automation
Hybrid Cloud Mesh
IBM Concert
IBM NS1 Connect
IBM Workload Automation & Workload Scheduler
Instana
Rapid Network Automation
SevOne
Turbonomic
User groups
Events
TechXchange Day
IBM TechXchange Conference
Upcoming IT Automation Events
IBM TechXchange Webinars
All IBM TechXchange Community Events
Participate
Gamification Program
Getting Started
Community Manager's Welcome
Post to Forum
Share a Resource
Share Your Expertise
Blogging on the IBM TechXchange Community
Connect with IT Automation Users
All IBM TechXchange Community Users
Resources
IBM TechXchange Group
IBM Champions
IBM Cloud Support
IBM Documentation
IBM Support
IBM Support 101
IBM Technology Zone
IBM Training
AIOps on IBM Z
TechXchange Day
Marketplace
Marketplace
IBM IT Automation Community
Come for answers, stay for best practices. All we're missing is you.
Ask a question
Join us for IBM TechXchange Day: AI and Automation
Skip main navigation (Press Enter).
Toggle navigation
Search Options
Participate
Blogs
Blog Viewer
AIOps
View Only
Group Home
Discussion
2K
Library
428
Blogs
627
Events
3
Members
6.4K
Share
Splunk Enterprise Integration with Cloud Pak for Watson AIOps
By
SREEDHAR KODALI
posted
Tue January 11, 2022 01:15 PM
3
Like
Co author: Eric Erpenbach
Introduction
IBM Cloud Pak for Watson AIOps is a scalable Ops platform that deploys advanced,
explainable AI across an IT Operations toolchain. It can reduce operational costs
significantly
by proactively assessing, diagnosing and resolving incidents emanating from infrastructure
and operations management before they become serious problems. It enables Site Reliability
Engineers (SREs) and CIOs to uncover hidden insights from mu
ltiple sources of data
–
like logs,
metrics and events
and delivers them directly into team collaboration tools like Slack,
Microsoft Teams, and ServiceNow in near real
-
time.
In version 3.2, IBM Cloud Pak for Watson AIOps provides a new AI analytics engi
ne that
delivers rich detection of issues with reduced noise. It also provides an easy training
oversight, along with the ability to manage all training data. It introduces statistical baselines
for log anomaly detection while improving the overall usabili
ty and scalability.
Data connections enable IBM Cloud Pak for Watson AIOps to collect log, metric and event
data from various incoming sources and communicate how day
-
to
-
day IT operations are
performing against a baseline model established by AI model tra
ining. Watson AIOps recognizes a
number of input sources for data integration such as ELK, Humio, LogDNA, Splunk, etc.
Splunk Integration
A Splunk input source can provide log and event data for Watson AIOps. Log data is used for
detecting anomalies, whereas event
data is used for processing events and alerts.
Splunk Enterprise enables searching, analyzing and visualizing data gathered from an IT
infrastructure. It indexes data from servers, applications, databases, network devices, and
virtual machines that are p
art of an IT infrastructure. It can collect data from different
infrastructure end points whether they reside locally, remotely or on cloud. Upon data
ingestion from log files and other sources, the received data is parsed and indexed so queries
can be m
ade against the indexed data.
Splunk Enterprise can be deployed either as a single instance or multiple instances. A single
instance deployment is useful for testing and evaluation purposes where multiple instances
deployment is necessary for production
needs. In a single instance deployment, all aspects of
processing data from input through indexing to search is handled by only one instance.
Splunk Enterprise employs forwarders to get data from remote machines. Splunk Universal
Forwarder is a forwarder
that requires minimal resources and resides on the machine where
data originates. It consumes remote data and sends it to Splunk Enterprise for indexing.
This blog post covers in detail the steps to integrate a single instance Splunk Enterprise
deployment
with IBM Cloud Pak for Watson AIOps for sourcing log data for anomaly
detection.
Integration Scenario
Figure1 below captures a simple integration scenario involving Splunk Enterprise 8.2.3
running on a standalone Red Hat 8.4 Linux VM and IBM Cloud
Pak for Watson AIOps 3.2
deployed on Red Hat OpenShift 4.7 cluster. A Splunk Universal Forwarder 8.2.3 deployed on
a second Red Hat 8.4 Linux VM forwards system logs to Splunk Enterprise instance running
on the first VM for indexing.
Let’s quickly see
various steps to deploy each one of these integration entities and connect
between them.
Splunk Standalone Instance
Follow the instructions laid out in
Splunk Enterpris
e official documentation to install version
8.2.3
on the first Red Hat 8.4 Linux VM.
To secure the Splunk platform, turn on HTTPS encryption for Splunk Web by creating a
/opt/splunk/etc/system/local/web.conf
file by setting
enableSplunkWebSSL
option to
true
and
then restart Splunk Enterprise instance:
Splunk Web interface (Figure 2) is available @
https://<splunk-standalone
-
hostname>:8000
with default
self
-
signed certificate:
Enable token authentication by logging to the Splunk Web with admin credentials and
then
create a new token for AIOps integration:
Select settings > tokens > enable token authentication > click New Token > (Figure 3) fill
-
in the
create token form fields appropriately, most notably the user, description and expiration >
click create butt
on > copy the text from the token field
To start monitoring localhost
/var/log
directory for events,
select settings > data inputs > files
& directories > click new local file & directory button > enter /var/log in the text box (Figure 4)
and then revie
w and submit the form
:
For receiving data from remote VMs into the Splunk Enterprise for indexing and querying, go
to
settings > forwarding and receiving
and then click on
+Add new
(Figure 5) link against
Configure receiving
under
Receive data
section
:
Enter the default port of
9997
to receive data from forwarders and click
save
button.
Splunk Forwarder
Next, login to the second Red Hat Linux m/c with admin credentials and install the matching
version of Splunk Universal Forwarder by following inst
ructions laid out in the official
documentation.
To configure forwarder to send data to the Splunk Enterprise standalone instance hosted on the first
Linux VM, run
splunk
command with
add forward
-
server
option by supplying
hostname:port
of the standalone hostname:
Next define inputs to the forwarder by invoking
splunk
command with
add monitor
option
and supplying system log directory such as
/var/log
:
Check
host
attribute field in the
search dashboard
of Splunk Web to verify whether newly
added forwarder host is
listed.
AIOps Analytics
Follow the
instructions
outlined in IBM Cloud Pak for Watson AIOps 3.2 knowledge center to
install Starter Profile of AI Manager on Red Hat OpenShift version 4.6 or above.
After AI Manager install is successfully done, login to the OpenShift cluster using
oc login
command on
the boot node:
To login to the AI Manager console, you need to retrieve admin password and URL to access
the AI Manager console:
Enter the URL in a browser to access the AI Manager console (Figure 6) and login with
admin
username and password retrieved above:
Splunk Connection
After logging into the AI Manager console, click
Data and tool connections
from either the left
hamburger menu under
Define
group or tile available on the main console page. Click
Add
connectio
n
and select
Logs category
and choose
Splunk add connection
entry:
This pops up a brief Splunk configuration section overview on the right side. Click
connect
button to proceed. In the
Add connection
details page, fill
-
in various connection entries
(Figure 8) appropriately with Splunk setup details gathered previously:
> Name: Connection name used for display; for ex, Splunk
-
Enterprise
-
Test
-
Connection
> Description: (optional) Describe the nature of this connection in few; for ex, Test Splunk
Enterpri
se integration with Watson AIOps
> Splunk service URL: URL for accessing REST API of the previously setup Splunk Enterprise standalone instance
; for ex,
https://<splunk
-
standalone-hostname>:8089
Note: 8089 is the API service port for Splunk Enterprise server
> Authentication type:
You can select either User ID/password or Token; preferably choose
Token type
> Token: Token value to access Splunk REST API; enter token retrieved during the Splunk standalone instance
setup
> Certificate: (optional) Certificate to verify SSL/TLS connection to Splunk
Note: Retrieve SSL certificate from the Splunk Enterprise server using openssl command
> Filters: (optional) Custom search query in Search Processing Language (SPL) to run on
Splunk index; leave this field blank
> Base parallelism: Number of search requests
that can be made in parallel; leave this field
to default value of 1
Note: Set base parallelism to something below maximum number of concurrent searches
possible with the above Splunk Enterprise setup. Refer to
/opt/splunk/etc/system/default/limits.conf fo
r further details.
> Sampling rate: The frequency at which data is sampled from Splunk in seconds for
processing live data; leave this field to default value of 60 seconds.
Click
Next
to proceed to optional Field mapping section (Figure 9)
–
you can leave
this field to
default for this simple integration scenario
Click
Test connection
button
to validate the field values entered. You should see
Test
succeeded
message with a
tick mark
.
Click
Next
to proceed to AI training and change risk scoring data sec
tion (Figure 10). Leave
Data flow off
for now. You can set mode to
Historical data for initial AI training
and turn on
Data flow when you are ready to undertake model training.
Click
Done
to complete the connection setup.
Conclusion
In this blog
post, we have covered simple integration scenario involving Splunk Enterprise
server, Splunk Universal forwarder and Cloud Pak for Watson AIOps and associated steps to
test the same. This would allow you to quickly setup Splunk for monitoring your IT
infra
structure and Watson AIOps for processing system logs in near real
-
time and providing
actionable insights.
References
For further details, refer to the following links:
(1) Install IBM Cloud Pak for Watson AIOps Starter Profile
–
https://www.ibm.com/docs/en/cloud
-
paks/cloud
-
pak
-
watson
-
aiops/3.2.0?topic=manager
-
starter
-
installation
(2) Creating Splunk Connections
–
https://www.ibm.com/docs/en/cloud
-
paks/cloud
-
pak
-
watson
-
aiops/3.2.0?topic=connections
-
splunk
(3) Install Splunk Enterprise
–
https://docs.splunk.com/Documentation/Splunk/8.2.3/SearchTutorial/InstallSplunk
(4) Install Splunk Universal Forwarder
–
https://docs.splunk.com/Documentation/Forwarder/8.2.3.1/Forwarder/Installanixuniversalfo
rwarder
0 comments
54 views
Permalink
IBM Community Home
Browse
Discussions
Resources
Groups
Events
IBM TechXchange Conference 2023
IBM Community Webinars
All IBM Community Events
Participate
Gamification Program
Community Manager's Welcome
Post to Forum
Share a Resource
Blogging on the Community
All IBM Community Users
Resources
Community Front Porch
IBM Champions
IBM Cloud Support
IBM Documentation
IBM Support
IBM Technology Zone
IBM Training
Marketplace
Marketplace
IT Automation
Topic groups
AIOps
Expertise Connect(EC) Group
Global IT Automation
Hybrid Cloud Mesh
IBM Concert
IBM NS1 Connect
IBM Workload Automation & Workload Scheduler
Instana
Rapid Network Automation
SevOne
Turbonomic
User groups
Events
TechXchange Day
IBM TechXchange Conference
Upcoming IT Automation Events
IBM TechXchange Webinars
All IBM TechXchange Community Events
Participate
Gamification Program
Getting Started
Community Manager's Welcome
Post to Forum
Share a Resource
Share Your Expertise
Blogging on the IBM TechXchange Community
Connect with IT Automation Users
All IBM TechXchange Community Users
Resources
IBM TechXchange Group
IBM Champions
IBM Cloud Support
IBM Documentation
IBM Support
IBM Support 101
IBM Technology Zone
IBM Training
AIOps on IBM Z
TechXchange Day
Marketplace
Marketplace
Powered by Higher Logic