Non responsive tomcat - http & jk pool 100%

[Kees de Kooter]

For the second time in a week my app (on tomcat-behind-apache) became totally unresponsive. The only solution I could find was kill -9 and restart.

I noticed that booth the http and the jk pool were measuring 100%. Heap memory usage was slowly climbing. Garbage collection were much less frequent.

What could cause the http pool to flll up? And is it possible to flush it somehow?

The apache error log displays the following:

[Sat Feb 27 02:36:13 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Sat Feb 27 02:36:13 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 127.0.0.1:8009 (localhost)

[kjkoster]

Dear Kees,

What would is if you could make a thread dump when Tomcat is in that state. Then you can analyze what the threads from the http and jk pools are doing. There are a number of short tutorials on this topic on Java-monitor.

I am a bit surprised that both these pools fill up. Does your Tomcat get traffic from more than one source? How come the HTTP pool fills up when you have an Apache to relay the traffic?

Kees Jan

[Kees de Kooter]

The http pool is used because two apps deployed in this server communicate over http. One app is serving content to the other.

Content is pulled in with an import tag, like so:
<c:import url="http://metronome.nl:8080/metronome/content/public-news-list.page">

This is probably not the most efficient way of sharing data between two webapps :(.

Could this be exhausting the http pool?

[kjkoster]

Dear Kees,

It may not be the most efficient way to share data, but at least it is in-line with the servlet spec. Plus, it will allow you to run in more than one Tomcat instance easily. Scalability, baby.

The fact that both these pools fill up helps debug the problem. Based on that I would wager that there is a bug in the HTTP-serving webapp, not the AJP serving one. I'd venture that you will find that the AJP serving threads blocked waiting for network I/O.

I am curious to see that thread dump. I have a feeling the answer is in there. :)

Kees Jan

[Kees de Kooter]

Dear Kees Jan,

I share your curiosity. I expect the next occurence next week.

In the meantime I will analyze the content serving code thoroughly.

[kjkoster]

Dear Kees,

You can take a thread dump when you see the HTTP pool starting to fill up. Perhaps you can take one now, and one later when the pool fill level is rising. No need to wait until you are using up all threads. Basically any thread that is not blocked on the pool is suspect.

Kees Jan