For context, I've built an IBM Operational Decision Manager (ODM) 8.0 infrastructure, in order that I can then go through the process to migrate from version 8.0 ( which is circa 2012 ) to the latest version, 8.9.
I'd built a Decision Center and a Decision Server, on a single VM, each in its own WebSphere Application Server (WAS) 8.0 cluster, with a shared DB2 server, hosting the requisite RTSDB and RESDB databases.
This was all working ....
Scroll forward a few weeks, and I've resumed the VM ( which was sleeping ), and am attempting to deploy a RuleApplication from Decision Center to Decision Server.
Alas, this is what I see in the Decision Center UI: -
and this is what I see in the Decision Center log - SystemOut.log : -
[05/07/17 08:26:14:956 BST] 00000063 Builder I Use of the Rule Common Engine...
[05/07/17 08:26:19:513 BST] 00000063 IlrRulesetArc I Ruleset archive built [4293ms]
[05/07/17 08:26:19:514 BST] 00000063 IlrRulesetArc I -------------------------------
[05/07/17 08:26:19:514 BST] 00000063 IlrRulesetArc I Checking ruleset archive (rule analysis enabled)...
[05/07/17 08:26:22:589 BST] 00000063 IlrRulesetArc I -------------------------------
[05/07/17 08:26:22:628 BST] 00000063 IlrRulesetArc I ====> Overall time: 7408ms
[05/07/17 08:26:22:588 BST] 00000063 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/DecisionCenterClusterMember1_1c2e535_17.07.05_08.26.21.8371698818595718579454.txt ilog.rules.factory.translation.IlrTranslationCompiler com.ibm.sample.Helper
[05/07/17 08:26:23:195 BST] 00000063 E Cannot find key 10004 for resource bundle com.ibm.rules.res.console.messages
java.util.MissingResourceException: Can't find resource for bundle com.ibm.rules.res.console.messages, key en_GB
at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:360)
The reference to the en_GB locale is a good possibility, as the default locale for Decision Center is en_US, but I'm fairly sure I changed it when I built it - and it wouldn't be getting en_GB from anywhere else .........
But, before I change anything, I should check the Decision Server log - SystemOut.log - which shows all sorts of errors: -
[05/07/17 08:22:25:405 BST] 00000016 SystemOut O 848 [server.startup : 1] ERROR ilog.rules.res.console.util.IlrConsoleInitializer - GBRXC0101E: Initialization failed.
ilog.rules.res.console.IlrConsoleException: Persistence initialization failed.
ilog.rules.res.persistence.IlrDAOException: Unexpected exception during the build of DAO "persistence".
ilog.rules.res.persistence.IlrDAOException: Cannot connect to the database.
com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][2043][11550][3.61.65] Exception java.net.ConnectException: Error opening socket to server odm.uk.ibm.com/192.168.153.132 on port 60,006 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499
which made me think.
When I resumed the VM, the Deployment Manager was running, but the Node Agent, Decision Center and Decision Server were all down - so I started them.
Then I noticed that DB2 wasn't running - I ordinarily check this with a simple netstat command: -
netstat -aon | grep LISTEN | grep 60006
as the DB2 instance is listening on port 60006.
This didn't show anything :-(
So I started the DB2 instance: -
su - db2inst1
db2start
*BUT* I didn't restart the JVMs.
To validate this, I stopped and then started the Decision Server cluster.
And, guess what, the RuleApp deploy from Decision Center to Decision Server just worked.
So, to summarise, I made no changes to my configuration, I merely ensured that DB2 was running BEFORE starting Decision Server.
Interestingly Decision Center didn't "care" that the DB wasn't up when it started - I guess it doesn't matter until one actually logs in and tries to use Decision Center.
So another problem solved :-)