Hi All,

Please anybody can help that I have shutdown the Integration Server. And now I not able to start the Integration Server.

Please find the below log that I have taken from the nohup.out log

Changing working directory to /opt/webMethods822/IntegrationServer
Starting as an application using Java because it is not set to use Tanuki Wrapper.
Debug options:
file:/opt/webMethods822/profiles/IS/bin/.options loaded
Registering Software AG Platform Hooks …
log4j:WARN No appenders could be found for logger (com.softwareag.platform.jaas.internal.Activator).
log4j:WARN Please initialize the log4j system properly.
[Journal Logger Initializer] INFO: Reading config root “com.softwareag.platform.logging.configroot”
[Journal Logger Initializer] INFO: Resolved config root to “com.softwareag.platform.logging.configroot”.
06-Jun-2017 16:56:25 com.webMethods.sc.config.ConfigurationLogger log
INFO: Out-of-band file changes will be monitored every ‘60000’ milliseconds
Persistence bundle starting…
Persistence bundle started.
06-Jun-2017 16:56:26 Log InfoDeviceManager started
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No installable units to add.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No IU properties to add.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No installable units to remove.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No IU properties to remove.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No installable units to move.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No extra requirements.
Starting Integration Server
06-Jun-2017 16:56:57 com.webMethods.sc.config.ConfigurationLogger log
INFO: Out-of-band file changes will be monitored every ‘60000’ milliseconds
!SESSION 2017-06-06 16:56:23.708 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.6.0_27
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=solaris, ARCH=sparc, WS=gtk, NL=en_GB

!ENTRY org.eclipse.osgi 2 0 2017-06-06 16:57:00.678
!MESSAGE Info: cycle(s) found while stopping bundles: [[com.ctc.wstx,javax.xml.stream]].
Persistence bundle stopping…
Persistence bundle stopped.
[p2] Tue Jun 06 16:57:01 BST 2017 - [System Bundle Shutdown] Unsaved preferences when shutting down org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager

Regards,
Sunil

Hi Sunil,

can you provide the content of /opt/webMethods822/IntegrationServer/logs/server.log?

The nohup.out is not pointing in the right direction currently.

Make sure that IS has been shutdown completely before restarting.
Sometimes shutdown is not completing.

What was the reason for shutting down the IntegrationServer?

Before restarting the next time rename the nohup.out and server.log so you will have a fresh one which just contains the informations about this restart try.

Regards,
Holger

Hi Holger,

I have attached the clean log of restart of application.

And
What was the reason for shutting down the IntegrationServer?
→ we have used update manager to upgrade the patch and after server went to down and going to be restart as showing in the command line and IS did not come to up.
and the moment I am trying to restart the server, I have found that system IS came up and went to down.

Could you please look into the log and suggest what we can do to start the server.

Regards,
Sunil
server.log (8.53 KB)

Hi Sunil,

please restore the config files from config/backup to config, esp. those from the jdbc directory structure.
Try to start the IntegrationServer.

If it still fails try the following:

Backup the following directories:

• IntegrationServer/db
• IntegrationServer/DocumentStore
• IntegrationServer/XAStore
• IntegrationServer/WmRepository4

Remove the following directories:

• IntegrationServer/db
• IntegrationServer/DocumentStore
• IntegrationServer/XAStore

Try to start again.

Please provide us the outcome of this.

Regards,
Holger

Hi Holger,

Sorry for the late response.

I have taken backup of folders that you said. And I tried start the application again but application is still down. I have attached the log for your reference.

Regards,
Sunil
server.log (36.5 KB)

Hi Holger,

Please find the server log again that has reporting extra information comparing to the last log that I have attached.

Regards,
Sunil

server.log (54.9 KB)

Hi Sunil,

from the logs I see the following:

1. SAP-Libraries are outdated:

2017-06-13 14:31:26 BST [SAP.0101.0026C] SAP JCo version: 3.0.12 (2014-11-21) JavaRfc 2.2.13
2017-06-13 14:31:26 BST [SAP.0101.0027C] SAP JCo Native Library version: 720.713
2017-06-13 14:31:26 BST [SAP.0101.0044C] SAP IDoc Library version: 3.0.11 (2014-06-20)

Recent versions are JCo 3.0.16 and IDoc Library 3.0.12.

2. Please check your Broker if it is up and running and that it is configured correctly.
   Broker should started BEFORE starting the IntegrationServer.

3. Please check connectivity to your BEA/Oracle WebLogic Server.

4. Did you apply some IS_Core Fixes lately?
   If so, you will have to apply DC_DBS_Fix and update your ISInternal Database Component by using the Database Configurator.
   This is related to the following message:
   [ISS.0137.0017E] Scheduler: Exception getting tasks from database: [sag-cjdbc42-0011][Oracle JDBC Driver][Oracle]ORA-00904: “RUN_AT”: invalid identifier

5. Is there another IS running on the same box?

2017-06-13 14:37:53 BST [ISP.0046.0012I] Enabling HTTP Listener on port 9999
2017-06-13 14:37:53 BST [ISS.0070.0013W] Failed to start listener HTTPListener@9999. The following error was encountered: Address already in use
2017-06-13 14:37:53 BST [ISP.0046.0012I] Enabling HTTP Listener on port 5555
2017-06-13 14:37:53 BST [ISP.0046.0012I] Enabling HTTP Listener on port 15006
2017-06-13 14:37:53 BST [ISS.0070.0013W] Failed to start listener HTTPListener@15006. The following error was encountered: Address already in use

If so, please change the Diagnostic and PRT Port.
This can be found in the following files:
IntegrationServer/config/server.cnf
IntgerationServer/packages/WmRoot/config/listeners.cnf
IntgerationServer/packages/WmPRT/config/listeners.cnf

Can you provide a list of installed Fixes from UpdateManager?

Regards,
Holger

Hi Holger,

Please find the applied patches:

Installed? Fix Name

1. [I] All fixes
   

2. [I]   Adapter Runtime
   

3. [I]     Adapter Runtime 8.2 SP2 Fix 7
   

4. [I]   Adapters
   

5. [I]     SAP Adapter 7.1 Fix13
   

6. [I]     JDBC Adapter 6.5 Fix 33
   

7. [I]     webMethods JMS Adapter 6.1 Fix 16
   

8. [I]   Broker
   

9. [I]     Broker_8.2.2_CSharpApi_Fix1
   

10. [I] Broker_8.2.2_Core_Fix2 (SOL)
11. [I] Broker_8.2.2_JMSApi_Fix1
12. [I] Common Library
13. [I] SCG_8.2SP2_Audit_Fix4
14. [I] SCG_8.2.2_Terracotta_Fix4
15. [I] ABE_CL_8.2_SP2_Fix11
16. [I] SCG_8.2_SP2_Entrust_Fix4
17. [I] SCG_8.2_SP2_Glue_Fix2
18. [I] Database Configuration
19. [I] DC_8.2_SP2_DBS_Fix15
20. [I] Integration Server
21. [I] DEP_8.2_SP2_Fix3
22. [I] Integration Server Core 8.2 Fix 23
23. [I] Flat File 8.2 SP2 Fix6
24. [I] IS_8.2_SP2_Tomcat6_Fix6
25. [I] Monitor UI
26. [I] Monitor MWS 8.2 SP1 Fix 6
27. [I] My webMethods Server
28. [I] MWS_8.2_SP1_Fix7
29. [I] Broker_8.2.1_Portal_Fix2
30. [I] SSX
31. [I] Fix 8.2.2.0003-0833 for SSX (SOL)
32. [I] Web Services Stack
33. [I] WSStack 8.2 SP2 Fix19
34. [I] webMethods Monitor
35. [I] Monitor 8.2.1 Fix 6

Regards,
Sunil

Hi Sunil,

did you run the database configurator after applying the DC_8.2_SP2_DBS_Fix15 to your installation?

As per the Readmes you are htting the issue PIE-36477.

PIE-36477 (IS_8.2_SP2_Core_Fix20)
After transitioning to or from daylight savings time, scheduled 
tasks run twice or not at all. 

Integration Server runs scheduled tasks based on the time relative 
to the time zone. This caused issues when entering and exiting 
daylight savings time, specifically tasks ran twice or not at all. 

Now, Integration Server runs scheduled tasks based on coordinated 
universal time (UTC). Because Integration Server runs the tasks 
without regard to the relative time zone, the start and end of 
daylight savings time does not affect the execution of scheduled tasks. 

Important! If you are using an external database for the ISInternal 
functional alias in Integration Server, after installing a fix that 
includes PIE-36477 and before starting Integration Server, make sure 
that the latest scripts for ISInternal have been applied to the 
database used by the JDBC pool alias associated with the 
ISInternal functional alias.  

Can you check your database schema for the view INSTALLED_COMPONENT (only lists latest version) or table COMPONENT_EVENT (lists all versions incl. migrations to newer versions) for the versions of the different database parts?

From your list of Fixes I miss the SCG_8.2_SP2_DataDirect4.2_Fix1.
But your log files indicate that you are running a DataDirect Driver 4.2 already.

You should check if it is possible for you to apply all latest Fixes for 8.2 to your environment as this will be neccessary anyway when planning a migration to a more recent version of webMethods.

May I suggest to migrate from JMS Adapter to the Built-In JMS services pub.jms:*?
This should work even with BEA/Oracle WebLogic Servers.

Regards,
Holger

Hi Holger,

The system Integration server came up. But the diagnostic port is showing as disabled and not able to enable the port and it is showing as:

Failed to start HTTPListener@9999: Address already in use

And schedulers also not displaying and throwing error as

[SoftwareAG][Oracle JDBC Driver][Oracle]ORA-00904: “RUN_AT”: invalid identifier

Could you please help on this as well.

Regards,
Sunil

Hi Sunil,

please read my previous post.

It has the informations for the RUN_AT issue (PIE-36477).

Do you have Optimize installed on the same box?
If so, shutdown InfrastructureDC as it is IntegrationServer-based.

Additionally you can change the Diagnostic Port to any other free port between 1025 and 65535.

Regards,
Holger

Hi Holger,

As you have told I have run the dbConfigurator with migrate and recreate actions.

Before running script the catalogue is:

•  Installed Components      *
  

• BLZ 10 Blaze WEBMDATA,WEBMINDX
• RUL 10 BusinessRules WEBMDATA,WEBMINDX
• CCS 25 Central Configuration WEBMDATA,WEBMINDX
• XRF 11 CrossReference WEBMDATA,WEBMINDX
• DSL 20 DistributedLocking WEBMDATA,WEBMINDX
• IDR 10 DocumentHistory WEBMDATA,WEBMINDX
• ISC 41 ISCoreAudit WEBMDATA,WEBMINDX
• ISI 40 ISInternal WEBMDATA,WEBMINDX
• MED 10 MediatorEvents WEBMDATA,WEBMINDX
• MWS 36 My webMethods Server WEBMDATA,WEBMINDX
• OPM 24 Operation Management WEBMDATA,WEBMINDX
• PRA 50 ProcessAudit WEBMDATA,WEBMINDX
• PRE 60 ProcessEngine WEBMDATA,WEBMINDX
• PRP 25 Reporting WEBMDATA,WEBMINDX
• PST 20 Staging WEBMDATA,WEBMINDX
• SRP 15 SuiteReporting WEBMDATA,WEBMINDX

and After it is showing as:

• Executing action
• Action : catalog
• Database : Oracle
• URL : jdbc:wm:oracle://10.143.1.122:1630;serviceName=WMDB
• User : wmadmin

•  Installed Components      *
  

• ACT 16 ActiveTransfer WEBMDATA,WEBMINDX
• CCS 25 Central Configuration WEBMDATA,WEBMINDX
• ISC 41 ISCoreAudit WEBMDATA,WEBMINDX
• ISI 40 ISInternal WEBMDATA,WEBMINDX
• OPM 24 Operation Management WEBMDATA,WEBMINDX
• PRA 50 ProcessAudit WEBMDATA,WEBMINDX
• PRE 60 ProcessEngine WEBMDATA,WEBMINDX

And I have gone through the tables and it seems some tables missed over there. Is it will make any problem and please suggest what all actions we have to perform.

Regards,
Sunil

Hi Sunil,

please can you explain for which components/products you have run the dbConfigurator?

Except for very rare cases it should not be neccessary to recreate any parts of the database schema.

I use separate schemas for different parts of the databases:

• MWS
• Trading Networks
• Optimize
• Archive (IS and BPM)
• IntegrationServer (IS, BPM, Reporting, Staging)
• …

This eases to determine which parts of database are affected.

When the tables for the MWS are lost your MWS should no longer be working.

Regards,
Holger

Hi Holger,

I have run the dbConfigurator for all schemas. Because after migrating schema also issue is still persisting.

And your words are correct that is MWS is not loading.

What is the next action to perform to get the tables that support MWS.

And now I am getting the new error as below for the schedulers:

[SoftwareAG][Oracle JDBC Driver][Oracle]ORA-00604: error occurred at recursive SQL level 1 ORA-01653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM ORA-02002: error while writing to audit trail ORA-00604: error occurred at recursive SQL level 1 ORA-01653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM

Please help.

Regards,
Sunil

Hi Sunil,

please contact your DBA to enhance quota settings or filesize for Tablespace SYS.

Auditlogging inside database has exceeded the available space in the tablespace file.

Regards,
Holger