Problem faced with Tomcat logs Catilina.out

[deepakgupta66]

WE have faced a problem with Tomcat logs catalina.out on our server. It suddenly starts taking lots of of space on server and unexpectedly increased the size of catlina.out to 94 GB within an hour. We were forced to empty this file manually.

I would like to know the possible cause for the same since we are not able to understand how it happens, was it an attack on Tomcat or Java or there is any security issue in Tomcat 5.0 or we may require to put some restrictions on usage of tomcat.

Please help me on finding out the solution for this unusual even happens on our server.

Deepak Gupta

[Kees de Kooter]

Hi Deepak,

Have you been able to take a look at the contents of the logfile? What kind of entries do you see?

Are your applications logging to their own log files or to stdout (i.e. catalina.out)? In the latter case your application log settings could cause it.

[deepakgupta66]

Hi Deepak,

Have you been able to take a look at the contents of the logfile? What kind of entries do you see?

Are your applications logging to their own log files or to stdout (i.e. catalina.out)? In the latter case your application log settings could cause it.

Hi Kees

We are not able to see the content of logfile because it was 94GB file and we were forced to delete it immi to keep our server running.

Our applications logging to stdout(i.e. catalina.out)

It is tomcat server common log file and our appliction (like web crawler not auto through user based)

My major issue is how this catlina.out used 94GB harddisk space within 30 minutes. We are using the same settings for years and this never happens.
What could be possible reason for this.

Deepak Gupta

[Kees de Kooter]

Hi Deepak,

You should really try to take a look at the contents of the logfile. On a unix system you can use tail for that, it only reads the last couple of lines.

Unless we know what is in the log we can only guess what is causing it.

[deepakgupta66]

Hi Deepak,

You should really try to take a look at the contents of the logfile. On a unix system you can use tail for that, it only reads the last couple of lines.

Unless we know what is in the log we can only guess what is causing it.

Hi Kees

I agree with you. We should have taken a look at log but unfortunately we have deleted that file imi because our server was completely chocked and having 0 B free sapce. due to reason all services were stopped. As immi action we have removed this file to keep server running. Anyway we can retrieve this file or some other logs can give us any hint about this activity.

We have chceked access logs, FTP logs etc but nothing exceptional found in those logs.

Another incident is when we have restarted server we found some critical files missing like shell and init , database files in mysql.

Deepak Gupta

[Kees de Kooter]

OK here are some more things you should check:
- is it possible youi deployed a new version of your application in which the log level is too low (i.e. debug level instead of info)?
- is it possible you are suffering form hardware failure, causing the log to get filled with error messages?
- is it possible your server is being infected with a virus or otherwise hacked?

[kjkoster]

Deepak,

If those files are missing, is the server hacked? Also, check your harddisks with smartctl to see if they have any hardware errors logged.

Kees Jan

[deepakgupta66]

Deepak,

If those files are missing, is the server hacked? Also, check your harddisks with smartctl to see if they have any hardware errors logged.

Kees Jan

Hi Kees

Thanks for your response. I am not able to analyse this Whether the server is hacked which is very rare since we are using IP based SSH only or it happens due to malfunctioning of some script since there are no log files available on server, all are deleted. Now I am looking for some tool which can recover these files. Please tell me if u are aware about any good tool which we can run on Linux server and can get these deleted files recovered.

[admin]

Dear Deepak,

You cannot recover files that were deleted from Linux file systems. That is what backups are for.

What I do is check the system configuration files that I change into an SVN repository. That way I can always rebuild a dead machine (and detect hacks). Likewise I put Tomcat and its webapps into an SVN repository.

Log files I never backup or try to recover in a crash. If they are lost, I just shrug and move on.

In your case, I would just restart the services and start monitoring disk fill levels. Personally, I use Zabbix for that, but there are dozens of excellent tools that can do that for you. That way, you will never be caught off-guard again.

Kees Jan

[deepakgupta66]

Dear Deepak,

You cannot recover files that were deleted from Linux file systems. That is what backups are for.

What I do is check the system configuration files that I change into an SVN repository. That way I can always rebuild a dead machine (and detect hacks). Likewise I put Tomcat and its webapps into an SVN repository.

Log files I never backup or try to recover in a crash. If they are lost, I just shrug and move on.

In your case, I would just restart the services and start monitoring disk fill levels. Personally, I use Zabbix for that, but there are dozens of excellent tools that can do that for you. That way, you will never be caught off-guard again.

Kees Jan

Thanks Kees. We are also doing the same. We have gone a head, installed new drive and start working. I want to analyse the cause of failure due to reason I was looking for these files.

Thanks for your suggestions.