PDA

View Full Version : Java Tomcat Debugging Help


Rabu
22-06-2009, 15:59
Technology in use

I have a 3rd party tax application (vertex) that is running on Java 1.6 within Tomcat 6.0. The above is installed on VM servers (4GB of memory) with window server 2003 OS. For high availability reasons we are using the F5 load balancing software where we create a VIP that points to the 2 backend VM web servers running the window OS referenced previously. Tomcat Java Heap Memory allocation is 1024 MB for the initial and max memory pool.

The tax application has a series of web services (lookup tax, tax quote, invoice request) that our host ERP system calls fairly heavily throughout the day and quite intentsively throughout the nightly invoicing and sales update processes (200,000 - 300,000 web service calls within a 2 hour window).


Problem
We are experiencing application issues from our host system to the vertex tax application where tomcat does not seem to go down but the process stops unexpectedly. If tomcat went down I would see a log entry correct?

Action taken to date

we have turned on the Tomcat access logs to track when the application is being hit. We also monitoring this log for the F5 logic where the load balancer performs an http get against port 8080 to ensure that tomcat is up before sending incoming traffic to either web server

I have installed JConsole on both web servers and pushed test web server loads through and the boxes seem to handle the incoming traffic fine.


Help Needed

I have very limited knowledge with respect to JConsole application testing.

Does anyone have a good knowledge reference for application testing with JConsole. Such as looks at these memory variables or focus your attention on Garbarage Collection, etc.

What is a good Java memory heap allocation for an application that is called 200,000- 300,000 times within a 2 hours window. Obviously application design impacts memory but I guess I'm wondering if the 1024 max memory setting is too low? The fact that tomcat does not go down makes me think that 1024 could be fine?

kjkoster
22-06-2009, 22:52
Dear Rabu,

First off, hat off for your excellent posting style and clear description. Impressive.

When you say "Tomcat does not seem to go down", do you mean that the Tomcat process is still running, just becomes unresponsive, or is the process also gone?

Restarts are indeed logged in catalina.out, provided you configured Tomcat to do so. If you are running with the default Tomcat configuration, restarts will indeed be logged.

When the Tomcat servers do not respond, what are their machines doing? Are their CPU's pegged at 100% load or idling around 0% load? Are the disks idle or chugging away?

In JConsole, look at the old/tenured memory pool. That pool should not go over (say) 85% and have a nice, flat sawtooth.

For the garbage collectors, you may want to consider installing Java-monitor and looking at the GC time graphs. That will tell you something about the time your application spends garbage collecting.

In JConsole, you can do this too, but you don't have a GC graph there. You'd have to look at the gc stats on the memory tab.

Rabu
26-06-2009, 15:33
At this point, I have run several tests with webservice call volumes of 100K and found that the GC old/tenured memory pool never reached 50%. The graphs looks good based on your feedback and other research that I've read.


Next Steps:

Is there any way to write JConsole results to a log?
Does J-Monitor offer this feature?

I'm hoping to establish a long term analysis for the application and currently I'm just able to view real time JConsole activity.

kjkoster
27-06-2009, 12:34
Dear Rabu,

Neither JConsole or Java-monitor offer any way to archive the measured data for more than a short period of time (jconsole: as long as it runs) (Java-monitor: two days). You can easily do this yourself, by writing some code that accessed JMX itself and stores the stats in a file or a database.

Here is a small piece of JMX coding for Java to get you started: http://java-monitor.com/forum/showthread.php?t=169

Kees Jan

PS. A future version of Java-monitor will offer long-term data storage and will let you see back for up to a year or maybe two. No ETA for this feature, though.