We are deploying our application on the AIX m/c using the Websphere application server 6.0. The issue is that unless the XDEUB option is specified the application server doesnot startup.

This is configured in the Application servers >  > Process Definition > Java Virtual Machine

Debug arguments:
-Djava.compiler=NONE -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=7777

Generic JVM arguments

-Dclient.encoding.override=iso-8859-1 -Dfile.encoding=iso-8859-1 -Djava.awt.headless=true -Djava.security.debug=failure -verbose:class -Xdebug -Dlogpath=test_server

Please advise on what needs to be modified so that the application server starts without the XDEBUG option.

This is just a suggestion / guess, but have you tried removing the debug arguments from the command and then started the JVM?  It's my understanding that debug arguments can only be passed into the JVM when the Debug Mode is Enabled.  If you disable the debug mode, perhaps the JVM doesn't like persisting the debug options...  Let us know.

Yes we have already tested with this option. If we remove the XDEBUG option the application server doesnot start up.

Understood from your initial post, but I was suggesting not including the arguments -Djava.compiler=NONE  -Xnoagent?  Thus your Debug Arguments would be "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777" (without the quotes) and not "-Djava.compiler=NONE -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=7777"

Yes that works fine. But the issue is with the XDEBUG in the Generic JVM arguments. When it is removed the application server doesnot start.

Please confirm that the JVM starts fine when the following is true:

Debug Arguments:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777

Generic JVM Arguments:

-Dclient.encoding.override=iso-8859-1 -Dfile.encoding=iso-8859-1 -Djava.awt.headless=true -Djava.security.debug=failure -verbose:class -Xdebug -Dlogpath=test_server

Also, you're looking to remove the -xdebug optiong from the GENERIC JVM ARGUMENTS too, correct? Do you need all of the arguments in the Generic JVM Arguments such as -Dlogpath? 

Yes with the following the JVM starts fine without any issue.

Debug Arguments:

 -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777

 Generic JVM Arguments:

 -Dclient.encoding.override=iso-8859-1 -Dfile.encoding=iso-8859-1 -Djava.awt.headless=true -Djava.security.debug=failure -verbose:class -Xdebug -Dlogpath=test_server

And i want to remove the -XDEBUG from the Generic JVM arguments.

The following is a required

-Dclient.encoding.override=iso-8859-1 -Dfile.encoding=iso-8859-1-Djava.security.debug=failure

Rest can be removed ...

Again, clarifying that attempting to start your JVM with ONLY the following is resulting in the JVM not starting completely:

Debug Arguments:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777

Generic JVM Arguments:

-Dclient.encoding.override=iso-8859-1 -Dfile.encoding=iso-8859-1-Djava.security.debug=failure

Yes the JVM doenot start.

 Can you please post part of the systemout.log showing the error?

:( Nothing is written in the SystemErr.log & the SystemOut.log.... that is problem.

Wow... that is definately a problem.  If there aren't logs of the failed start I'm stumped...  It might be time to open a PMR with IBM....