by Eric Montagnon
ITCAM for Transactions, with ISM agent, was installed and running on Linux system with ITM 6.3 fp6. Then ITM 6.3 fp7 was installed and ITM framework for the 32bit architecture was upgraded as well to 6.3 fp7.
Now ISM agent doesn't start, or start is incomplete and agent node never appears as online in TEP.
Symptoms :
1) agent doesn't start. Using command :
'itmcmd agent start is'
fails with
Starting ITCAM for Transactions: Internet Service Monitoring ...
*** Error in `/opt/IBM/ITM/lx8263/is/platform/linux2x86/bin/kisagent':
free(): invalid pointer: 0x095e0608 ***
======= Backtrace: =========
/usr/lib/libc.so.6(+0x7571d)[0x55c3071d]
/usr/lib/libstdc++.so.6(_ZdlPv+0x1f)[0x56e0197f]
/usr/lib/libstdc++.so.6(_ZNSs4_Rep10_M_destroyERKSaIcE+0x1b)[0x56e68ecb]
/opt/IBM/ITM/lx8263/is/../../li6243/gs/lib/libgsk8cms.so
(_ZN9GSKStringD1Ev+0x6b)[0x57947b8b]
/opt/IBM/ITM/li6243/gs/lib/libgsk8ssl.so(+0xcfab9)[0x57575ab9]
/opt/IBM/ITM/li6243/gs/lib/libgsk8ssl.so
(gsk_attribute_set_buffer+0x3da4)[0x5753ee94]
2) agent does start. using command :
'itmcmd agent start is'
'ps -ef | grep kis' command shows "kisagent" process is running, but ISM node never appears online in TEP
3)
(58A310EB.0016-1:kdebenc.c,364,"ssl_provider_constructor") TLS 1.0 protocol enabled
(58A310EB.0017-1:kdebenc.c,393,"ssl_provider_constructor") TLS 1.1 protocol enabled
(58A310EB.0018-1:kdebenc.c,416,"ssl_provider_constructor") TLS 1.2 protocol enabled
(58A310EB.0019-1:kbbssge.c,72,"BSS1_GetEnv") KDEBE_KEY_LABEL=GSK_KEY_LABEL="IBM_Tivoli_Monitoring_Certificate"
(58A310EB.001A-1:kbbssge.c,72,"BSS1_GetEnv") KDEBE_KEYRING_FILE=GSK_KEYRING_FILE="/opt/IBM/ITM/keyfiles/keyfile.kdb"
(58A310EB.001B-1:kbbssge.c,72,"BSS1_GetEnv") KDEBE_KEYRING_STASH=GSK_KEYRING_STASH="/opt/IBM/ITM/keyfiles/keyfile.sth"
4) For reference here is what cinfo reports, just after I upgraded my agent Linux system from 6.3.0.6 to 6.3.0.7 :
PC PRODUCT DESC PLAT VER BUILD INSTALL DATE
57 Monitoring Agent for IT_SSLCheck lx8266 06.31.00.01 201612201254 20161220 1457
84 Monitoring Agent for IT_IPRoute lx8266 06.22.00.00 - 20050823 2253
89 Monitoring Agent for IT_CommonOutScript lx8266 06.30.00.03 201612201256 20161220 1502
ax IBM Tivoli Monitoring Shared Libraries lx8263 06.23.05.00 d4099a 20161124 1341
ax IBM Tivoli Monitoring Shared Libraries lx8266 06.30.07.00 d6350a 20170214 1206
gs IBM GSKit Security Interface li6243 07.40.47.00 d4029a -
gs IBM GSKit Security Interface lx8266 08.00.50.69 d6276a -
is ITCAM for Transactions: Internet Service Monitoring lx8263 07.40.01.10 0327 20161124 1343
jr Tivoli Enterprise-supplied JRE lx8266 07.09.50.00 201609291010 -
lz Monitoring Agent for Linux OS lx8266 06.30.07.00 62801 20170214 1206
r4 Agentless Monitoring for Linux Operating Systems lx8266 06.30.07.00 201610181737 20170214 1206
ue Tivoli Enterprise Services User Interface Extensions lx8266 06.30.07.00 d6350a 20170214 1206
ui Tivoli Enterprise Services User Interface lx8266 06.30.07.00 d6350a 20170214 1206
then I used the command to upgrade ax and gs 32bit components to 6.3 fp7 :
./install.sh -h /opt/ISM74 -q -p /opt/eric/ITM6307/6.3.0-TIV-ITM_TMV-Agents-FP0007/unix/tflx8263.txt
now cinfo reports this :
PC PRODUCT DESC PLAT VER BUILD INSTALL DATE
57 Monitoring Agent for IT_SSLCheck lx8266 06.31.00.01 201612201254 20161220 1457
84 Monitoring Agent for IT_IPRoute lx8266 06.22.00.00 - 20050823 2253
89 Monitoring Agent for IT_CommonOutScript lx8266 06.30.00.03 201612201256 20161220 1502
ax IBM Tivoli Monitoring Shared Libraries lx8263 06.30.07.00 d6350a 20170214 1215
ax IBM Tivoli Monitoring Shared Libraries lx8266 06.30.07.00 d6350a 20170214 1206
gs IBM GSKit Security Interface li6243 08.00.50.69 d6276a -
gs IBM GSKit Security Interface lx8266 08.00.50.69 d6276a -
is ITCAM for Transactions: Internet Service Monitoring lx8263 07.40.01.10 0327 20161124 1343
jr Tivoli Enterprise-supplied JRE lx8266 07.09.50.00 201609291010 -
lz Monitoring Agent for Linux OS lx8266 06.30.07.00 62801 20170214 1206
r4 Agentless Monitoring for Linux Operating Systems lx8266 06.30.07.00 201610181737 20170214 1206
ue Tivoli Enterprise Services User Interface Extensions lx8266 06.30.07.00 d6350a 20170214 1206
ui Tivoli Enterprise Services User Interface lx8266 06.30.07.00 d6350a 20170214 1206
Origin of the problem:
6.3 fp7 "breaks" ISM agent.
Possible solutions:
1) from a valid non upgraded yet ISM agent linux system, backup the whole content of gskit v7, located in
$> Cd li6243
$> tar -cvpf gs-v7.tar ./gs
then restore it on the broken ISM agent system
$> Cd li6243
$> rm -rf gs (to remove gsk8)
$> tar -xvpf gs-v7.tar (to restore gsk7)
This has proven to work with RHEL V6, but not with RHEL V7 though.
2) uninstall ISM agent from the current ITM home directory where other ITM components were laid down, and install ISM agent into a separate directory, with 6.3fp7 in use.
3) IBM development team worked on a more long term solution, under APAR # IV93927
Remark1 : So far, this issue was seen only on Linux OS, not AIX or Windows.
Remark2 : ISM 7.4 (kisagent, nco_m_bridge, and nco_m_* monitors) all work with GSKit 8 as long as the framework is updated to 6.3.x (except FP7).
For example, ISM agent works with v6.3.5.0 framework with GSKit 8 (8.0.50.36). ISM agent with ITM 6.3.3.0 framework and GSKit 8.0.50.20 also works correctly:
ax IBM Tivoli Monitoring Shared Libraries
lx8263 Version: 06.30.03.00
gs IBM GSKit Security Interface
li6243 Version: 08.00.50.20
is ITCAM for Transactions: Internet Service Monitoring
lx8263 Version: 07.40.01.02
From
APAR # IV93927 is included in 7.4.0.0-TIV-CAMIS-fp0002
it's available for download from IBM Fix Central web site, see its readme :
https://www-01.ibm.com/support/docview.wss?rs=0&uid=isg400003438
4) Same issue impacts all other ITCAM for Transactions agents, like TO, TU, T3, etc, see :
https://www.ibm.com/developerworks/community/blogs/TransactionsFP01/entry/ITM_6_3_Fixpack_7_on_Linux_causes_ITCAM_agents_to_crash_on_startup?lang=en