Troubleshoot - unexpected shutdown of tomcat 5.5

[jerry]

I have deployed my web application (using struts) in tomcat webapps directory. Now I want to run this tomcat server continuously, but it doesn't. 0nce or twice a month it shutdown by itself. I have checked the log file and it is not showing any error. Does any one has idea why it is happening and what should I do to improve it.
Thanks

[Kees de Kooter]

Install the java-monitor probe and see if it runs out of resources.

[kjkoster]

Dear Jerry,

A little more information may be nice too: operating system, how you keep tomcat daemonized, Tomcat installation method.

Does tomcat restart when the machine is rebooted? Do the tomcat outages coincide with machine reboots?

Kees Jan

[jerry]

hello kjkoster,

I have tomcat 5.5 installed on linux (x86_64-pc-linux-gnu) server. Its binary was downloaded from tomcat's website (http://tomcat.apache.org/). It is running properly, but it automatically shutdown 2-3 times in a month then we have to restart it. We wanted it to run continuously as we have deployed our web application on it. It is not linked with machine reboot because this linux server is always running.

Sorry for replyig so late.

Thanks

[kjkoster]

Dear Jerry,

Well, Tomcat does not just shut down all by itself. I heard some anecdotes of people who found usage of System.exit() in the code of a web application. Calling System.exit() will indeed terminate the JVM.

If you have access to the Java source code of your web app, check for uses of System.exit(). If you do not have the code, or if you just want to search quickly, you can use findbugs to detect uses of System.exit() (http://java-monitor.com/forum/showthread.php?t=200). Don't forget to check the JSP pages in your webapp, since findbugs cannot check those.

Also, check that the server boot times and your Tomcat stop times. You can use the UNIX command uptime(1) (http://www.freebsd.org/cgi/man.cgi?query=uptime) to check when the server was last booted.

Of course, do as Kees suggests and install the Java-monitor monitoring probe (java-monitor.com/install.html). You'll receive an e-mail when the server goes off-line so at least you will know when to restart it. :-)

Kees Jan

[Barry]

Jerry,

Just another question... are you running on physical hardware or virtualization?
In case of physical hardware, do you have swap enabled (can you share the output of free -m)

Often these kinds of strage VM deaths can be the result of the OOM killer when you have exhausted all the memory in the machine, it starts with the biggest process first (which would most likely be Tomcat)

You should also check the Tomcat working directory to see if there are any VM dump files which could indicated the VM crashed. (Though oddly enough, if the VM dies from a native exception it leaves a lot in your log)

[jerry]

Hello Kees Jan,

I have checked whole application there is no System.exit() and also I checked server running status using update() and it has been running since 19 days, so this is also not problem because I got the same problem again yesterday.

I have installed java-monitor probe, so lets see if it'll return any thing.

Thanks

[Barry]

Cool,

Let us know.

Also keep in mind that if you are running on a virtual server like OpenVZ / Virtuozzo that they can decide to kill bigger processes when they run out of memory.

[jerry]

I found some errors in my log file and am feeling might be this is one of the reason that tomcat shutdowns:

1. Caused by: com.mysql.jdbc.CommunicationsException: The last packet successfully received from the server was40524 seconds ago.The last packet sent successfully to the server was 40524 seconds ago, which is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.

2. Caused by: java.net.SocketException: Broken pipe

I have checked on various posts and people are saying this is because usually waiting time on sql is 28,000 (which is true for my system) and after that it shut down. I don't know how to fix this. Have any idea of these errors.

Thanks again!!

[jerry]

Ok now I am getting results from java-monitor probe, but I didn't find any error in its statistics. I wanted to send those graphs, but I don't know how I can. Any other suggestions guys. But probe is really a good tool and sends an message when ever server stops.

[kjkoster]

Dear Jerry,

Here is a discussion on how to make the connection between Tomcat and MySQL more reliable (http://java-monitor.com/forum/showthread.php?t=70). Perhaps that might be of help?

Kees Jan

[jerry]

Hi Kees Jan,

Is it possible that java-monitior probe shows server is stopped and application is running? Actually here it is showing tomcat is not running but I can see my application is running. I have checked in my catalina.sh, it is not showing any error msg (just thread dump I don't know from where it came suddenly).

Thanks

[kjkoster]

Dear Jerry,

Unfortunately, the Java-monitor probe is not perfect, so there are some situations where the probe may show that your server is off-line, when really your server is working. For example, during a release of the collector service, all the probes are forced off-line (yes we will have redundant collector servers soon).

The probe should automatically retry after a couple of minutes and resume sending data.

The best thing to do is to check the maintenance announcements (http://java-monitor.com/forum/forumdisplay.php?f=4), where we post after each upgrade or update. If the probe had not resumed sending data in 15 minutes, check the logs to see what is wrong.

Are you sure that you did not cause that thread dump yourself with kill -3 or jstack or something?

Kees Jan

[jerry]

I was trying kill -3 yesterday, so it might possible it generated then and I didn't notice that time. In this thread dump, I can see lots of wait() thread and very few runnable threads. And I think I can understand why there are so many wait threads because I have set following attributes (in bold) in Resource tag.

<Resource name="jdbc/informDB" auth="Container" type="javax.sql.DataSource"
maxActive="100" maxIdle="30" maxWait="10000" factory="org.apache.commons.dbcp.BasicDataSourceFactory"
username="id" password="Pass" driverClassName="com.mysql.jdbc.Driver"
testWhileIdle="true" timeBetweenEvictionRunsMillis="10000" minEvictableIdleTimeMillis = "60000"
validationQuery="SELECT 1 = 1" logAbandoned="true"
url="jdbc:mysql://localhost/dataBase?autoReconnect=true"/>

Am I right? If so then what should be the ideal max number for these attributes?

Thanks a lot for helping so much.