CAM

ITCAM4Tx - Linux ISM Agent Does Not Start with ITM 6.3 fp7

By IMWUC Community Team posted Tue October 15, 2019 02:30 AM

  

by Eric Montagnon

ITCAM for Transactions, with ISM agent,  was installed and running on Linux system with ITM 6.3 fp6. Then ITM 6.3 fp7 was installed and ITM framework for the 32bit architecture was upgraded as well to 6.3 fp7.

 

Now ISM agent doesn't start, or start is incomplete and agent node never appears as online in TEP.

 

Symptoms :

1) agent doesn't start. Using command :

'itmcmd agent start is'

fails with

Starting ITCAM for Transactions: Internet Service Monitoring ...
*** Error in `/opt/IBM/ITM/lx8263/is/platform/linux2x86/bin/kisagent':
free(): invalid pointer: 0x095e0608 ***
======= Backtrace: =========
/usr/lib/libc.so.6(+0x7571d)[0x55c3071d]
/usr/lib/libstdc++.so.6(_ZdlPv+0x1f)[0x56e0197f]
/usr/lib/libstdc++.so.6(_ZNSs4_Rep10_M_destroyERKSaIcE+0x1b)[0x56e68ecb]
/opt/IBM/ITM/lx8263/is/../../li6243/gs/lib/libgsk8cms.so
(_ZN9GSKStringD1Ev+0x6b)[0x57947b8b]
/opt/IBM/ITM/li6243/gs/lib/libgsk8ssl.so(+0xcfab9)[0x57575ab9]
/opt/IBM/ITM/li6243/gs/lib/libgsk8ssl.so
(gsk_attribute_set_buffer+0x3da4)[0x5753ee94]

 

 

2) agent does start. using command :

'itmcmd agent start is'

'ps -ef | grep kis' command shows "kisagent" process is running, but ISM node never appears online in TEP

 

3)

(58A310EB.0016-1:kdebenc.c,364,"ssl_provider_constructor") TLS 1.0 protocol enabled
(58A310EB.0017-1:kdebenc.c,393,"ssl_provider_constructor") TLS 1.1 protocol enabled
(58A310EB.0018-1:kdebenc.c,416,"ssl_provider_constructor") TLS 1.2 protocol enabled
(58A310EB.0019-1:kbbssge.c,72,"BSS1_GetEnv") KDEBE_KEY_LABEL=GSK_KEY_LABEL="IBM_Tivoli_Monitoring_Certificate"
(58A310EB.001A-1:kbbssge.c,72,"BSS1_GetEnv") KDEBE_KEYRING_FILE=GSK_KEYRING_FILE="/opt/IBM/ITM/keyfiles/keyfile.kdb"
(58A310EB.001B-1:kbbssge.c,72,"BSS1_GetEnv") KDEBE_KEYRING_STASH=GSK_KEYRING_STASH="/opt/IBM/ITM/keyfiles/keyfile.sth"

 

4) For reference here is what cinfo reports, just after I upgraded my agent Linux system from 6.3.0.6 to 6.3.0.7 :
PC   PRODUCT DESC                                              PLAT    VER           BUILD           INSTALL DATE

57   Monitoring Agent for IT_SSLCheck                      lx8266  06.31.00.01   201612201254    20161220 1457
84   Monitoring Agent for IT_IPRoute                           lx8266  06.22.00.00   -               20050823 2253
89   Monitoring Agent for IT_CommonOutScript          lx8266  06.30.00.03   201612201256    20161220 1502
ax   IBM Tivoli Monitoring Shared Libraries          lx8263  06.23.05.00   d4099a          20161124 1341
ax   IBM Tivoli Monitoring Shared Libraries                   lx8266  06.30.07.00   d6350a          20170214 1206
gs   IBM GSKit Security Interface                            li6243  07.40.47.00   d4029a          -
gs   IBM GSKit Security Interface                                 lx8266  08.00.50.69   d6276a          -
is   ITCAM for Transactions: Internet Service Monitoring       lx8263  07.40.01.10   0327            20161124 1343
jr   Tivoli Enterprise-supplied JRE                                lx8266  07.09.50.00   201609291010    -
lz   Monitoring Agent for Linux OS                                lx8266  06.30.07.00   62801           20170214 1206
r4   Agentless Monitoring for Linux Operating Systems lx8266  06.30.07.00   201610181737    20170214 1206
ue   Tivoli Enterprise Services User Interface Extensions lx8266  06.30.07.00   d6350a          20170214 1206
ui   Tivoli Enterprise Services User Interface                lx8266  06.30.07.00   d6350a          20170214 1206


then I used the command to upgrade ax and gs 32bit components to 6.3 fp7 :
./install.sh -h /opt/ISM74 -q -p /opt/eric/ITM6307/6.3.0-TIV-ITM_TMV-Agents-FP0007/unix/tflx8263.txt

now cinfo reports this :
PC   PRODUCT DESC                                              PLAT    VER           BUILD           INSTALL DATE

57   Monitoring Agent for IT_SSLCheck                      lx8266  06.31.00.01   201612201254    20161220 1457
84   Monitoring Agent for IT_IPRoute                           lx8266  06.22.00.00   -               20050823 2253
89   Monitoring Agent for IT_CommonOutScript         lx8266  06.30.00.03   201612201256    20161220 1502
ax   IBM Tivoli Monitoring Shared Libraries         lx8263  06.30.07.00   d6350a          20170214 1215
ax   IBM Tivoli Monitoring Shared Libraries                  lx8266  06.30.07.00   d6350a          20170214 1206
gs   IBM GSKit Security Interface                           li6243  08.00.50.69   d6276a          -      
gs   IBM GSKit Security Interface                                lx8266  08.00.50.69   d6276a          -      
is   ITCAM for Transactions: Internet Service Monitoring  lx8263  07.40.01.10   0327            20161124 1343
jr   Tivoli Enterprise-supplied JRE                               lx8266  07.09.50.00   201609291010    -      
lz   Monitoring Agent for Linux OS                               lx8266  06.30.07.00   62801           20170214 1206
r4   Agentless Monitoring for Linux Operating Systems  lx8266  06.30.07.00   201610181737    20170214 1206
ue   Tivoli Enterprise Services User Interface Extensions lx8266  06.30.07.00   d6350a          20170214 1206
ui   Tivoli Enterprise Services User Interface               lx8266  06.30.07.00   d6350a          20170214 1206

 

Origin of the problem:

6.3 fp7 "breaks" ISM agent.

 

Possible solutions:

1) from a valid non upgraded yet ISM agent linux system, backup the whole content of gskit v7, located in

$> Cd li6243

$> tar -cvpf gs-v7.tar ./gs

 

then restore it on the broken ISM agent system

$> Cd li6243

$> rm -rf gs (to remove gsk8)

$> tar -xvpf gs-v7.tar (to restore gsk7)

This has proven to work with RHEL V6, but not with RHEL V7 though.

 

2) uninstall ISM agent from the current ITM home directory where other ITM components were laid down, and install ISM agent into a separate directory, with 6.3fp7 in use.

 

3) IBM development team worked on a more long term solution, under APAR # IV93927

 

Remark1 : So far, this issue was seen only on Linux OS, not AIX or Windows.

 

Remark2 : ISM 7.4 (kisagent, nco_m_bridge, and nco_m_* monitors) all work with GSKit 8 as long as the framework is updated to 6.3.x (except FP7).
For example, ISM agent works with v6.3.5.0 framework with GSKit 8 (8.0.50.36).  ISM agent with ITM 6.3.3.0 framework and GSKit 8.0.50.20 also works correctly:
ax      IBM Tivoli Monitoring Shared Libraries
         lx8263  Version: 06.30.03.00
gs      IBM GSKit Security Interface
         li6243  Version: 08.00.50.20
is      ITCAM for Transactions: Internet Service Monitoring
         lx8263  Version: 07.40.01.02

From

APAR # IV93927 is included in 7.4.0.0-TIV-CAMIS-fp0002

it's available for download from IBM Fix Central web site, see its readme :

https://www-01.ibm.com/support/docview.wss?rs=0&uid=isg400003438

4) Same issue impacts all other ITCAM for Transactions agents, like TO, TU, T3, etc, see :

https://www.ibm.com/developerworks/community/blogs/TransactionsFP01/entry/ITM_6_3_Fixpack_7_on_Linux_causes_ITCAM_agents_to_crash_on_startup?lang=en

0 comments
2 views

Permalink