Hi,

Recently I got some errors on Operations consoles which is failed te get data to be shown:

All services are unreachable and EngMonApp appears not running:

1- I restarted all AppWatcher services : the Issue persists

2- I restarted all DS Services : Engine, Agent and Metadataserver : The issue persists

3- a lot of amount of JITDUMP,CORE,JAVACORE and Snap created in differents locations : DSODB/bin , ASBNode and ASBNode/bin , DSEngine.

4- I looked into the javacore files and I found there is many of lines with this following message (error) :

1TISIGINFO Dump Event "gpf" (00002000) received

All clients and jobs continue to work normally, only watchers and operations consoles services not working normally : the console is down

Please do you have an idea about that?

The version is 11.3.1 on an AIX 7.1 and the java version is :

1CIJAVAVERSION JRE 1.7.0 AIX ppc64-64 build (pap6470_27sr1-20140411_01(SR1) )

Thanks in advance,

BEst regards,

Hi,

you could check SystemOut.log and SystemErr.log files of a WebSphere Application Server that executes an Operation Console application.

should find a java process.

Hi yuri3,

Thanks for your replay.

I have checked those 2 files and this is what I found:

1-SystemErr.log:

0000004e SystemErr R Checking /tmp/informationServer/Reportingmdbsvm1011Node01server1/engine/JREPORT/history ... OK

0000004e SystemErr R log4j:WARN No appenders could be found for logger (jet.jetc.util.JetCManager).

0000004e SystemErr R log4j:WARN Please initialize the log4j system properly.

0000004e SystemErr R log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

2-SystemOut.log:

I got this

[10/20/20 16:02:02:090 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1272529 seconds.

[10/20/20 16:07:02:261 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1272829 seconds.

[10/20/20 16:12:02:499 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1273130 seconds.

[10/20/20 16:17:02:729 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1273430 seconds.

[10/20/20 16:22:03:339 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1273730 seconds.

[10/20/20 16:27:04:003 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1274031 seconds.

[10/20/20 16:32:04:262 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1274331 seconds.

[10/20/20 16:37:04:431 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1274632 seconds.

[10/20/20 16:42:04:577 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1274932 seconds.

[10/20/20 16:47:04:751 CEST] 000000b3 AgentChecker W Handshake has not been established to the ASBAgent at NACUUS2145.BU.SAM:31531 for over 1275232 seconds.

when I stopped all services (DSEngine, Agent Node, ...) I got logs like the following:

[10/20/20 16:49:04:864 CEST] 000000c2 FFDCJanitor I com.ibm.ws.ffdc.impl.FFDCJanitor doCleanupIfNeeded FFDC0004I: FFDC log file management removed 36 of 36 files that have reached their configured maximum

age

[10/20/20 16:49:04:919 CEST] 000000c2 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /s2idts/was/V8.5/profiles/InfoSphere/logs/ffdc/server1_8c0e3a_20.10.20_

16.49.04.8723503136639146024392.txt com.ibm.ws.spi.uow.UOWManagerImpl.runUnderNewUOW 175

[10/20/20 16:49:04:934 CEST] 000000c2 IIS-DSTAGE E IIS-DSTAGE-0: REST operation threw an internal server exception

com.ibm.datastage.rest.exceptions.DataStageRESTServiceOperationFailedException: Service operation failed when performing action 'services' for resource 'engines/NACUUS2145.BU.SAM'

at com.ibm.datastage.rest.servlets.actions.ServicesAction.getDetailedStatus(ServicesAction.java:134)

at com.ibm.datastage.rest.servlets.actions.ServicesAction.doObjectOperation(ServicesAction.java:93)

.........

Caused by: java.rmi.RemoteException: com.ibm.wsspi.uow.UOWException: javax.transaction.RollbackException: Global transaction timed out after 7200 seconds

........

Caused by: javax.transaction.RollbackException: Global transaction timed out after 7200 seconds

.....

; nested exception is:

com.ibm.iis.isf.j2ee.transaction.J2EETransactionException: com.ibm.wsspi.uow.UOWException: javax.transaction.RollbackException: Global transaction timed out after 7200 seconds

.....

Caused by: javax.transaction.RollbackException: Global transaction timed out after 7200 seconds

.....

Caused by: com.ibm.iis.isf.j2ee.transaction.J2EETransactionException: com.ibm.wsspi.uow.UOWException: javax.transaction.RollbackException: Global transaction timed out after 7200 seconds

....

Caused by: javax.transaction.RollbackException: Global transaction timed out after 7200 seconds

.....

[10/20/20 16:49:04:963 CEST] 000000c2 IIS-DSTAGE E IIS-DSTAGE-0: Internal error occured while processing REST API URL GET '/ibm/iis/ds/api/engines/NACUUS2145.BU.SAM/services' from IP '125.30.114.28' for user 'duadm' (HTTP status=500). HttpSessionId is: '<none>'. Elapsed time was 1,277,413,453ms

Thanks in advance,

BEst regards,

You could also check ASBAgent log files

asbagent_startup.err

asbagent_startup.out

Hi Yuri3

In the asbagent_startup.err log file I find this message repeated many times:

DFT 2020: [./Agent.sh] The Agent stopped abnormally with error 160

In asbagent_startup.out log file I find the following on many lines:

[./Agent.sh] Restarting the Agent...

[./Agent.sh] Retryable value from the Agent was returned: 160

Best regards,