kjkoster
29-10-2009, 12:20
Dear All,
Sometimes Tomcat won't shut down without you killing the process by unfriendly means. The standard advise given in such cases is that you can use thread dumps to diagnose this problem. But how does one actually do that?
Before we can dive into diagnosing this issue, we need to have a look at Tomcat's shutdown sequence and what might cause it to hang. As you know, the jvm shuts down when there are no live non-daemon threads left. Tomcat runs as a collection of threads, as does your application. When you ask Tomcat to shut down, it terminates all of its threads. The jvm sees that all regular threads have died and shuts itself down.
This process may be upset when applications start their own threads. To be precise, this process may be upset when applications do not properly clean up all of the threads they started. If an application leaves a thread running, the jvm will wait for it to terminate. It will wait for any amount of time. It has all the time in the world. That may cause Tomcat's shutdown appear to hang. It is not deadlocked or broken. The jvm is just patiently waiting for the last thread to finish.
As I said in my previous post regarding thread dumps, you need to define what you are looking for before you start. Thread dumps are just too large to root through at random. In the case of analysing hanging processes on shutdown, that question will be "which thread is the jvm waiting for?"
With that in mind, send the shutdown command to your Tomcat server. Wait for a few seconds to give Tomcat some time to clean up. Then use kill -3 or jstack to get a thread dump.
I find it convenient to save thread dumps in a separate file. That allows me to let grep and awk loose on the dump. They allow me to make sense easily. Here is the command I use, where 1234 is the process id of my Tomcat server.
% jstack 1234 > dump.txt
If you never looked at a thread dump before, I suggest you open the file in a text editor and just read it for a few minutes. See if you can make sense of it. In particular, see if you can see what state xxxx each thread is in. You need that later, when you try to resolve your hanging problem.
Let's make a list of all live threads, shall we?
% cat dump.txt | fgrep tid=
Don't you wish you had given your threads useful names? :-)
There are broadly four categories of threads: JVM threads, platform threads, library threads and your own threads. Of these categories, you can ignore the JVM threads (e.g. "SIGINT handler", "CompilerThread1") and the platform threads. Our platform is Tomcat, so we van ignore any thread that was started by Tomcat (e.g. "http-127.0.0.1-8080-Processor1").
Some library threads you can also ignore. For example the Derby database starts a thread named "derby.rawStoreDaemon" and one named "derby.antiGC". You have to check that your code closes and cleans up the database resources correctly. Once that is ok you can safely ignore these threads when doing this analysis.
Threads that you started should all have been killed at this stage. You have to start and stop them properly. A simple test that I sometimes use is to check that no threads with my code in the stack trace exist:
% cat dump.txt | fgrep com.example
at com.example.HogServlet$1.run(HogServlet.java:18)
And there is the culprit. Looking at the code we can see it was started, but not stopped. The proper way to use threads on a servlet engine is not to use the servlet constructor or any other way, but to use the init() and destroy() methods.
That is all. All that is left to do is to fix the code, deploy a new version and enjoy the ease of stopping Tomcat.
Oh, and if you have the time: give all threads you start a descriptive name. That way it is *much* easier to find threads that should have been dead.
Kees Jan
Sometimes Tomcat won't shut down without you killing the process by unfriendly means. The standard advise given in such cases is that you can use thread dumps to diagnose this problem. But how does one actually do that?
Before we can dive into diagnosing this issue, we need to have a look at Tomcat's shutdown sequence and what might cause it to hang. As you know, the jvm shuts down when there are no live non-daemon threads left. Tomcat runs as a collection of threads, as does your application. When you ask Tomcat to shut down, it terminates all of its threads. The jvm sees that all regular threads have died and shuts itself down.
This process may be upset when applications start their own threads. To be precise, this process may be upset when applications do not properly clean up all of the threads they started. If an application leaves a thread running, the jvm will wait for it to terminate. It will wait for any amount of time. It has all the time in the world. That may cause Tomcat's shutdown appear to hang. It is not deadlocked or broken. The jvm is just patiently waiting for the last thread to finish.
As I said in my previous post regarding thread dumps, you need to define what you are looking for before you start. Thread dumps are just too large to root through at random. In the case of analysing hanging processes on shutdown, that question will be "which thread is the jvm waiting for?"
With that in mind, send the shutdown command to your Tomcat server. Wait for a few seconds to give Tomcat some time to clean up. Then use kill -3 or jstack to get a thread dump.
I find it convenient to save thread dumps in a separate file. That allows me to let grep and awk loose on the dump. They allow me to make sense easily. Here is the command I use, where 1234 is the process id of my Tomcat server.
% jstack 1234 > dump.txt
If you never looked at a thread dump before, I suggest you open the file in a text editor and just read it for a few minutes. See if you can make sense of it. In particular, see if you can see what state xxxx each thread is in. You need that later, when you try to resolve your hanging problem.
Let's make a list of all live threads, shall we?
% cat dump.txt | fgrep tid=
Don't you wish you had given your threads useful names? :-)
There are broadly four categories of threads: JVM threads, platform threads, library threads and your own threads. Of these categories, you can ignore the JVM threads (e.g. "SIGINT handler", "CompilerThread1") and the platform threads. Our platform is Tomcat, so we van ignore any thread that was started by Tomcat (e.g. "http-127.0.0.1-8080-Processor1").
Some library threads you can also ignore. For example the Derby database starts a thread named "derby.rawStoreDaemon" and one named "derby.antiGC". You have to check that your code closes and cleans up the database resources correctly. Once that is ok you can safely ignore these threads when doing this analysis.
Threads that you started should all have been killed at this stage. You have to start and stop them properly. A simple test that I sometimes use is to check that no threads with my code in the stack trace exist:
% cat dump.txt | fgrep com.example
at com.example.HogServlet$1.run(HogServlet.java:18)
And there is the culprit. Looking at the code we can see it was started, but not stopped. The proper way to use threads on a servlet engine is not to use the servlet constructor or any other way, but to use the init() and destroy() methods.
That is all. All that is left to do is to fix the code, deploy a new version and enjoy the ease of stopping Tomcat.
Oh, and if you have the time: give all threads you start a descriptive name. That way it is *much* easier to find threads that should have been dead.
Kees Jan