Sunday, July 25, 2010

When sendsignal and Ctrl+break fail to generate Java thread dump in Windows

I handled a customer incident where it was looked like the application was hang. As our application is running as NT service - I suggested to use sendsignal in order to generate threads dump file. this worked fine in my local environment but on the customer environment the file was not generated. I then send a batch file that launch the application as console application instead than NT service and suggested to press Ctrl+Break in the application window in order to generate threads dump file.
This also failed.
The customer was using Remote Desktop application. I suspect that the failure is related to the fact that the "Ctrl+Break" was done from remote so I suggested to do it on the server's keyboard itself but this also failed.

Next, I thought to generate the threads dump from inside the application. I googled on this but did not find away. I consult a colleague of mine on this and he told me on com.ibm.jvm.Dump.JavaDump() see "IBM JVM 6.0 diagnostics guide".
Before implementing this into our application, I went to consult my team lead on this idea - she thought there must be a more standard way to generate threads dump. She did not want to add new functionality (dump generation) that was not necessary to our application activity. She asked me to try and find if such standard way exist.
She was right - there is such way and it worked for the customer.
I found it in the diagnostics guide above and it involves the use of three steps:
  1. generating system dump
  2. extracting JVM info from the system dump
  3. viewing the thread's stacks and monitors
How to generate system dump in windows
  1. install "userdump" from Microsoft site: http://www.microsoft.com/downloads/en/confirmation.aspx?familyId=e089ca41-6a87-40c8-bf69-28ac08570b7e&displayLang=en
  2. run "userdump -p" . it will list all running processes and their process id in the right column
  3. find the application process id
  4. run "userdump pid " . this should generate " process_name.dmp" file in "C:\kktools\userdump8.1\x86" folder (in case you are using x86 platform)

How to extract the JVM info from the system dump

  1. cd to C:\kktools\userdump8.1\x86
  2. run "jextract process_name .dmp". this should generate "process_name.dmp.zip" and "process_name.dmp.xml"

How to view the thread's stacks and monitors

  1. copy the "process_name.dmp.zip" to a local folder
  2. open CMD.exe
  3. cd to the local folder
  4. run "jdmpview -zip process_name.dmp.zip"
  5. execute "set logging file file_name"
  6. execute "set logging on"
  7. execute "info thread *"
  8. open the log file in editor and investigate



By the way, jdmpview has rich functionality and you can read on it in the diagnostics guide.