Hi,

I have a two node was-cluster. Since last two three days, I am not able to syncronise these two nodes .

If I reboot my server, then after manually starting the deployment manager (./startManager.sh), it gets started but when i try to stop it using ./stopManager.sh . it says:

ADMU0509I: The Deployment Manager "dmgr" cannot be reached. It appears to be stopped.

But actually it is started and i can access the admin console. Please request you to kindly update on this as it is urgent.

Thanks..

do you see the PID file under logs directory ?

Dear Joseph thanks for the reply,

yes I can see the pid file (dmgr.pid) under the logs/dmgr directory
(/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr ) .

Please suggest what should be done?
 

Hi ankit,

  can you check if the process is running?
 
    cat /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr/dmgr.pid
    
    and then
 
    ps -ef|grep PIDNUMBER

  If the process is running i suggest to kill
 
    kill -9 PIDNUMBER
    
  and start again dmgr
 
   /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/bin/./startManager.sh
 
   Then try login again in console if you can not you need to have some error in SystemOut.log (or in FFDC files)
   
   Put here to give more suppport
   
   Hope hits helps.
   
 Kind regards

 ankit,
 
   Do you have security enabled?
   
 regards

Hi Gabriel,

Thanks for your prompt reply. 

I did as suggested by you. I was able to successfully start the dmgr manager (startManager) and login to the console after killing the pid.

But when I tried to stop it ( stopManager). It gives me following error:

 

[root@rea-apps01 bin]# ./startManager.sh
ADMU0116I: Tool information is being logged in file
/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr/startServer.log
ADMU0128I: Starting tool with the rea-apps-dm01 profile
ADMU3100I: Reading configuration for server: dmgr
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3000I: Server dmgr open for e-business; process id is 12112


[root@rea-apps01 bin]# ./stopManager.sh
ADMU0116I: Tool information is being logged in file
/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr/stopServer.log
ADMU0128I: Starting tool with the rea-apps-dm01 profile
ADMU3100I: Reading configuration for server: dmgr
ADMU0509I: The server "dmgr" cannot be reached. It appears to be stopped.
ADMU0211I: Error details may be seen in the file:
/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr/stopServer.log
[root@rea-apps01 bin]#


Please see the last 80 lines of stopserver log:

at org.apache.soap.rpc.Call.invoke(Unknown Source)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$4.run(SOAPConnectorClient.java:316)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:301)
... 32 more

[10/1/13 15:16:59:019 IST] 0000000a AdminTool A ADMU0509I: The server "dmgr" cannot be reached. It appears to be stopped.
[10/1/13 15:16:59:020 IST] 0000000a AdminTool A ADMU0211I: Error details may be seen in the file: /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr/stopServer.log
************ Start Display Current Environment ************
Host Operating System is Linux, version 2.6.9-22.ELsmp
Java version = J2RE 1.5.0 IBM J9 2.3 Linux x86-32 j9vmxi3223-20080315 (JIT enabled)
J9VM - 20080314_17962_lHdSMr
JIT - 20080130_0718ifx2_r8
GC - 200802_08, Java Compiler = j9jit23, Java VM name = IBM J9 VM
was.install.root = /opt/ibm/was61/rea-apps01
user.install.root = /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01
Java Home = /opt/ibm/was61/rea-apps01/java/jre
ws.ext.dirs = /opt/ibm/was61/rea-apps01/java/lib:/opt/ibm/was61/rea-apps01/classes:/opt/ibm/was61/rea-apps01/lib:/opt/ibm/was61/rea-apps01/installedChannels:/opt/ibm/was61/rea-apps01/lib/ext:/opt/ibm/was61/rea-apps01/web/help:/opt/ibm/was61/rea-apps01/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/properties:/opt/ibm/was61/rea-apps01/properties:/opt/ibm/was61/rea-apps01/lib/startup.jar:/opt/ibm/was61/rea-apps01/lib/bootstrap.jar:/opt/ibm/was61/rea-apps01/lib/j2ee.jar:/opt/ibm/was61/rea-apps01/lib/lmproxy.jar:/opt/ibm/was61/rea-apps01/lib/urlprotocols.jar:/opt/ibm/was61/rea-apps01/java/lib/tools.jar
Java Library path = /opt/ibm/was61/rea-apps01/java/jre/bin:/opt/ibm/was61/rea-apps01/bin::/usr/lib
Current trace specification = *=info
************* End Display Current Environment *************
[10/1/13 16:29:19:001 IST] 0000000a ManagerAdmin I TRAS0017I: The startup trace state is *=info.
[10/1/13 16:29:19:065 IST] 0000000a AdminTool A ADMU0128I: Starting tool with the rea-apps-dm01 profile
[10/1/13 16:29:19:067 IST] 0000000a AdminTool A ADMU3100I: Reading configuration for server: dmgr
[10/1/13 16:29:21:130 IST] 0000000a JMXClassLoade W Could not find tmx4jTransform.jar in /opt/ibm/was61/rea-apps01/etc/tmx4jTransform.jar - Interoperability to older versions of WebSphere is disabled
[10/1/13 16:29:21:218 IST] 0000000a SSLConfig W CWPKI0041W: One or more key stores are using the default password.
[10/1/13 16:29:21:225 IST] 0000000a SSLConfigMana I CWPKI0027I: Disabling default hostname verification for HTTPS URL connections.
[10/1/13 16:29:21:858 IST] 0000000a WSKeyStore E CWPKI0033E: The keystore located at "/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/etc/trust.p12" failed to load due to the following error: Unable to verify MAC..
[10/1/13 16:29:21:872 IST] 0000000a WsServerStop E ADMU3002E: Exception attempting to process server dmgr
[10/1/13 16:29:21:874 IST] 0000000a WsServerStop E ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host rea-apps01 at port 8879 with SOAP connector security enabled.
[10/1/13 16:29:21:875 IST] 0000000a WsServerStop A ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host rea-apps01 at port 8879 with SOAP connector security enabled.
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:486)
at com.ibm.ws.management.tools.LaunchUtils.getAdminClient(LaunchUtils.java:372)
at com.ibm.ws.management.tools.LaunchUtils.getAdminClient(LaunchUtils.java:312)
at com.ibm.ws.management.tools.WsServerStop.runTool(WsServerStop.java:332)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:264)
at com.ibm.ws.management.tools.WsServerStop.main(WsServerStop.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:183)
at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:90)
at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:72)
at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActivator.java:78)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:92)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:68)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:400)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:336)
at org.eclipse.core.launcher.Main.basicRun(Main.java:280)
at org.eclipse.core.launcher.Main.run(Main.java:977)
at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:336)
at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:91)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:67)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:522)
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:341)
... 26 more
Caused by: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.SocketException; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.SocketException]
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:343)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.(SOAPConnectorClient.java:176)
... 31 more
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.SocketException; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.SocketException]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown Source)
at org.apache.soap.rpc.Call.invoke(Unknown Source)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$4.run(SOAPConnectorClient.java:316)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:301)
... 32 more

[10/1/13 16:29:21:882 IST] 0000000a AdminTool A ADMU0509I: The server "dmgr" cannot be reached. It appears to be stopped.
[10/1/13 16:29:21:883 IST] 0000000a AdminTool A ADMU0211I: Error details may be seen in the file: /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr/stopServer.log



 

 

Hi Gabriel,

If you are talking about administrative and application security, then yes it is enabled.

Https is also configured.

firewall and selinux is also disabled.

thanks,
Ankit

 

Anit,

  You need to stop using user/pwd how has management permisions. Try with one how has acccess to AdminConsole.

 
/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/bin/./stoptManager.sh -username USERNAME -password PASSWORD

regards,

Ankit,

  I addition you have the next errors.
 
[10/1/13 16:29:21:858 IST] 0000000a WSKeyStore E CWPKI0033E: The keystore located at "/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/etc/trust.p12" failed to load due to the following error: Unable to verify MAC..
[10/1/13 16:29:21:872 IST] 0000000a WsServerStop E ADMU3002E: Exception attempting to process server dmgr
[10/1/13 16:29:21:874 IST] 0000000a WsServerStop E ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host rea-apps01 at port 8879 with SOAP connector security enabled.
[10/1/13 16:29:21:875 IST] 0000000a WsServerStop A ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host rea-apps01 at port 8879 with SOAP connector security enabled.


  pic.dhe.ibm.com/infocenter/wasinfo/v6r1/...

"Unable to verify MAC." error when the wrong keystore password is used

You might receive the following error when the keystore password is not being used correctly.

CWPKI0033E: The keystore located at "C:/WebSphere/AppServer/profiles/AppSrv01/etc/trust.p12"
            failed to load due to the following error: Unable to verify MAC.

Change the Password field that references this keystore by using the correct password. The default password is WebAS. Never use this password in a producti

  you have some problem with your /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/etc/trust.p12 password....
 
  can you try to open with iKeyman or keytool /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/etc/trust.p12 to verify the password?
 
regards

Hi Gabriel,

I tried your method also but it also failed,

Please review the output for your reference::

[root@rea-apps01 bin]# ./stopManager.sh -username admin -password XXXXX
ADMU0116I: Tool information is being logged in file
/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr/stopServer.log
ADMU0128I: Starting tool with the rea-apps-dm01 profile
ADMU3100I: Reading configuration for server: dmgr
ADMU0509I: The server "dmgr" cannot be reached. It appears to be stopped.
ADMU0211I: Error details may be seen in the file:
/opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/logs/dmgr/stopServer.log

Regards,
 

I have changed my last reply review please.

The error is with trust.p12 file password

regards

Hi Gabreil,

our  configuration is like on one  node we have both the Application server and deployment manager and on other node we have the second application server.

Deployment manger admin console url :

https:// ab.cdf.1.251:9043/ibm/console/logon.jsp

server1 host file:

[root@rea-apps01 bin]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
ab.cdf.1.246 apps01.irctcnorth.com apps01
ab.cdf.1.247 apps02.irctcnorth.com apps02
ab.cdf.1.251 rea-apps01

I can ping the nodes using their IPs as well as Hostnames both ways. i can also connect to the nodes using ssh (using hostname).

 

Dear Gabriel,

When i executed  the keytool command it shows the following:

[root@rea-apps01 bin]# keytool -list -v -keystore /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/etc/trust.p12
keytool error: java.io.IOException: Invalid keystore format

 

Try with

keytool -list -v -keystore /opt/ibm/was61/rea-apps01/profiles/rea-apps-dm01/etc/trust.p12 -storetype PKCS12

then type keystore password to verify that is correct

Hi Gabriel,

The Application server is not starting now.It failed to initialise.

Looking at the latest logs that came in, the SystemOut is reporting    
following errors.                                                      
.                                                                      
Caused by: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4  vmcid:    
0x49421000  minor code: 56  completed: No                              
 at                                                                    
com.ibm.ws.orbimpl.transport.WSTransport.createListener(WSTransport.java
:859)                                                                  
 at                                                                    
com.ibm.ws.orbimpl.transport.WSTransport.initTransports(WSTransport.java
:597)                                                                  
 at                                                                    
com.ibm.rmi.iiop.TransportManager.initTransports(TransportManager.java:1
50)          
 at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1221)                
 at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1681)               
 at org.omg.CORBA.ORB.init(ORB.java:364)                               
 at com.ibm.ws.orb.GlobalORBFactory.init(GlobalORBFactory.java:86)     
 at com.ibm.ejs.oa.EJSORBImpl.initializeORB(EJSORBImpl.java:179)       
 at com.ibm.ejs.oa.EJSServerORBImpl.(EJSServerORBImpl.java:102)  
 at com.ibm.ejs.oa.EJSORB.init(EJSORB.java:55)                         
 at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:407)       


Kindly help.

Thanks

Ankit,

  have you change any DNS or Hostname? this type of errors ocurrs when WAS can not create some listener port on host, because it cannot resolve the host.
 
  You need to have some ffdc file saying that "host XXXX cannot be resolved"
 
  Review your SystemOut.log maybe before the error is saying why cannot create listener or your ffdc log files.
 
  By the way, have you been able to check your trust.p12 password? (my last answer in this thread)
 
  Hope this helps.  Tell us if you need more support.
 
regards

sometimes you can have mulitple dmgrs running on same server so I usually do a

 ps -ef | grep WsServer | grep -v grep | grep java | awk {'print $1,$2,$NF,$(NF-3)'} | grep dmgr 

fake output example
{owner} {pid} {last field} {config file(last field -3)}
wasadmin 2594 dmgr /opt/IBM/WebSphere6.0/AppServer/profiles/Dmgr01/config


now if you have only one and you see its still running you can kill it  

 kill -9 2594

now it is posible that the you have file permisions issues too.. where  you have a mix of ownership.. 
 
root 2594 dmgr /opt/IBM/WebSphere6.0/AppServer/profiles/Dmgr01/config

Where

ls -lt /opt/IBM/root/AppServer/profiles/Dmgr01/logs/dmgr -
 -rw-r--r-- 1 root root       4 yyyy-mm-dd hh:mm dmgr.pid

Websphere has a utility to help with permisions and ownership..

called chutils.sh

it located in the instutils folder under the was root dir

see link for more info.. USE with caution.. pic.dhe.ibm.com/infocenter/wasinfo/v8r0/...

  
 

hi,

The issue got resolved. Someone from the client site had deleted the etc directory from was_home in which one jar file exists.

I copied it from the other site and gave required permissions and then the isssue got resolved.

Thanks for all your support.