1) What is your exact level of IBM Liberty Profile ND? On what platform?
2) What web server type are you using? What version? On what platform?
3) What are the symptoms of the "problem"?
- What is the URL that the user is trying to access?
- What error message does the user see in the browser?
- At what time did the problem happen?
4) Enable access to the /server-status page of the web server.
The Apache mod_status servlet must be enabled and access to the /server-status URL must be enabled in your http configuration file.
5) Set the Plug-in LogLevel to "Trace":
Edit the plugin-cfg.xml file and set the LogLevel to Trace, like this:
LogLevel="Trace"
6) Set the Intelligent Management trace specification to "DEBUG":
Add the following lines to the server.xml of one of your Liberty collective controllers:
<dynamicRouting>
<traceSpecification name="default" specification=":DEBUG" />
</dynamicRouting>
Run the command to generate a new plugin-cfg.xml.
See: Dynamic Routing Liberty collective command actions
Alternatively, you can edit the existing plugin-cfg.xml file, and add the following inside the <IntelligentManagement> xml stanza:
<TraceSpecification name="default" specification=":DEBUG"/>
7) Copy the newly generated plugin-cfg.xml file from the Liberty server directory to the Plugins config directory on the web server. If possible, please stop and restart the web server so the trace will show everything from the beginning.
8) Capture a trace of the Dynamic Routing services on the Liberty collective controller:
Add the following to the server.xml of the Liberty collective controller servers:
<logging traceSpecification="*=audit:com.ibm.ws.dynamic.routing.*=debug:com.ibm.ws.collective.routing.*=debug"/>
9) At the time of the problem, capture additional information:
Capture a "dump" of the Liberty collective controller that the web server is communicating with:
Generating a Liberty server dump from the command linehttp://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_setup_dump_server.html
If you have a replica set configured for the Collective Controllers, please perform the above dump command for each controller to ensure a consistent view of the Liberty Collective
NOTE: The following mbean commands apply only to Liberty 16.0.0.4 versions and earlier. These commands to generate the Collective repository dump do not apply to Liberty versions newer than 16.0.0.4
On the "WebSphere:feature=dynamicRouting,type=DynamicRouting,name=DynamicRouting" MBean invoke the “dumpPostResponse” method with the arguments ("/<output_directory>/DynamicRoutingResponse.txt”.””).
See Interface DynamicRoutingMBean
On the "WebSphere:feature=collectiveController,type=RoutingInfoManager,name=RoutingInfoManager" MBean, invoke the “dump” method with the arguments ("/<output_directory>/RoutingInfo.txt”,””).
Here are example curl commands that can be run to invoke these MBeans. If you do not have curl available, you can use the jconsole command on the collective controller host to invoke the MBeans.
curl --user <controller_userName> --data '{"params":[{"value" : "/<output_directory>/DynamicRoutingResponse.txt", "type" : "java.lang.String"},{"value" : "","type" : "java.lang.String" }],"signature":["java.lang.String","java.lang.String"] }' --header "Content-Type:application/json" --request POST --insecure "https://localhost:
curl --user <controller_userName> --data '{"params":[{"value" : "/<output_directory>/RoutingInfo.txt", "type" : "java.lang.String"},{"value" : "","type" : "java.lang.String" }],"signature":["java.lang.String","java.lang.String"] }' --header "Content-Type:application/json" --request POST --insecure "https://localhost:
Replace <controller_userName> with a user that has administrative access to the collective. The command will prompt for the password.
Replace <output_directory> with a filesystem location with enough free space to hold the generated information. The full information should only be in the megabytes, so tmp would probably be appropriate.
Replace <controller_httpsPort> with the httpsPort of the collective controller listed as STARTED in the /server-status page.
Each curl command will cause a file to be written to the output directory. The two files are
DynamicRoutingResponse.txt
RoutingInfo.txt
Examine the output of the /server-status page in the web server to determine the collective controller that the web server is currently communicating with. This is the collective controller that is listed as STARTED under “connectors” on the /server-status page. Save the /server-status page as an html file to send to IBM support.
10) After capturing the traces of the problem, please send the answers to the questions above and these files to IBM Support:
- Web server configuration and log files:
- httpd.conf
- error_log
- access_log
- Plug-in configuration and log files:
- plugin-cfg.xml
- http_plugin.log
- Output of the /server-status page (saved as an html file)
- zip file the "dump" of the Liberty collective controller
For Liberty versions 16.0.0.4 and earlier:
- files created by running the curl commands in step 9:
- DynamicRoutingResponse.txt
- RoutingInfo.txt
- Liberty collective controller configuration and log files:
- server.xml
- zip of the entire logs directory and subdirectories
11) After the tracing has been completed, change Plug-in Level back to "Error", and remove the DEBUG trace specification from the Intelligent Management section. Also turn off the Liberty Collective traces by setting the trace specification back to *=info only.
NOTE: keep an eye on the size of the http_plugin.log file. It does NOT wrap like WAS log files. So, if the plug-in log is growing very large (over 2 GB) you will need to stop the web server, and delete or rename the file, then restart the web server. OR, you may want to consider using RotateLogs to automatically rotate the plug-in log to a new file based on time or size.