You have finally set up your own server and your application using this guide, and you feel proud that your app is now running in the vast web. A few days later, when you accessed your application’s URL, you are greeted with a Server Not Responding error! In panic, you SSH into your server to see what is going on, and soon find out that you have just run out of disk space.

What happened?

Most likely the reason is that your log files grew larger and larger as your application is running and it started eating through the entire disk. A couple of possible reasons are:

• There is a background process that runs database queries every set amount of time, and those queries get logged
• Using a background queue such as DelayedJob that continuously logs debug messages into the log file
• Malicious scrapers and bots that repeatedly attempt to gain admin access to your application (by using common login paths such as those used in WordPress, etc)
• Unnecessary logging of events that have large contents (e.g. logging entire objects or entire request data)

To prevent this scenario we need a way to make sure our logs are cleaned up automatically, so we don’t need to manually remove old logs as they have a tendency to be forgotten. One popular tool for this is a program called logrotate.

Logrotate works by making sure that your log files don’t grow in size unchecked. It compresses the log file and labels it with a timestamp so you can go back to older log files, while keeping the current log file contents only within the specified time range. For example, it can compress and archive logs at the end of the day every day so you begin with a fresh log file at the start of each day.

Installation

Let’s start by installing it, here we assume that we are using Ubuntu or any Debian-based system.

sudo apt-get update
sudo apt-get install logrotate

System Configuration

Logrotate’s configuration file can be found in /etc/logrotate.conf. Modify it using your preferred text editor.

sudo nano /etc/logrotate.conf

By default you will see something like this:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
# compress

This is the system or overall setting of logrotate. It is not recommended that we actually modify this file (although you can if you wish) and it is best practice to use specific configuration files for each log file we want to rotate/archive.

Custom Configuration

The custom configuration files are placed in the /etc/logrotate.d directory. We create a separate file for each log type so it is easier to modify and manage. For example, here is a configuration for rotating Nginx logs:

Start by creating the custom configuration file called “nginx”

sudo nano /etc/logrotate.d/nginx

The contents of the file looks like:

/opt/nginx/logs/access.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
copytruncate
}

/opt/nginx/logs/error.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
copytruncate
}

• daily – specifies how often you want the logs to be archived (can be daily, weekly, etc). I recommend setting it to “daily” especially in a production environment as it keeps the archives easier to handle.
• rotate 30 – keep the last N archives of the log. Since we have set it to “daily”, it will keep up to 30 days worth of archived logs. You can set this as high as you want as long as the disk usage is reasonable. You can also set it to the last 10 days or 2 weeks if the logs get bigger.
• compress – the archived logs are compressed using gzip (recommended). This keeps the file size much lower than raw logs.
• delaycompress – the previous archive will not be compressed until the next archiving schedule. For example, the log archive for yesterday will not be compressed until tomorrow. This is useful especially if you have programs that continuously write contents into your log file while it is being archived. This ensures that the program can still write into a log file as that file remains uncompressed.
• missingok – if any log file cannot be found, just search for the next log file in the configuration. This ensures that the logrotate program does not exit unexpectedly in case one log file is missing.
• notifempty – if the log file is empty, do not archive it. This is important as you only keep a certain number of archives, and this makes sure that you don’t have archived empty files that will push out older archived entries.
• copytruncate – when a log is archived, logrotate copies the contents of the log file into another file (with a timestamp). This option then tells logrotate to just remove/truncate the copied entries from the original log file. As with delaycompress, this option is needed when programs continuously write into the log file and this option ensures that the same log file is being used by the program to prevent it from exiting unexpectedly (due to it unable to access the log file).

Another explanation can be found in this StackOverflow question:

Please note that copytruncate makes a backup copy of the current log and then clears the log file for continued writing. The alternative is to use create which will perform the rotation by renaming the current file and then creating a new log file with the same name as the old file. I strongly recommend that you use copytruncate unless you know that you need create. The reason why is that Rails may still keep pointing to the old log file even though its name has changed and they may require restarting to locate the new log file. copytruncate avoids this by keeping the same file as the active file.

Application Configuration

We can now set up our application-specific configuration for rotating our application logs. This is done in a similar fashion as the sample Nginx configuration above by creating a separate file for your application configuration and specifying the location of our log files.

sudo nano /etc/logrotate.d/app_name

This example assumes that the common deploy configuration is located in a “shared” folder, as is the case when using Capistrano.

/var/www/app_name/shared/log/production.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
dateext
copytruncate
su deploy deploy
}

The configuration parameters are similar to the sample configuration for Nginx, except for the following:

• dateext – the archived log files will be appended with the date when it is processed. The default format is YYYYMMDD. This makes searching for archived logs easier.
• su deploy deploy – logrotate will archive the logs using the specific user and group to prevent issues with permissions. In this example the user is “deploy” and the group is “deploy”.

We can also use a wildcard when specifying the location of our log files so we can include everything in the directory without specifying each log one by one:

/var/www/app_name/shared/log/*.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
dateext
copytruncate
su deploy deploy
}

Running Logrotate Automatically

Once logrotate has been configured, you will want it to automatically run every day instead of running it manually. To do this, we will set up a daily cron job that will do this for you. If you installed logrotate using a package manager, the daily cron job will be automatically added as well (or you can set it up manually if not). This job is located in:

/etc/cron.daily/logrotate

#!/bin/sh

# Clean non existent log file entries from status file
 cd /var/lib/logrotate
 test -e status || touch status
 head -1 status > status.clean
 sed 's/"//g' status | while read logfile date
 do
 [ -e "$logfile" ] && echo "\"$logfile\" $date"
 done >> status.clean
 mv status.clean status

test -x /usr/sbin/logrotate || exit 0
 /usr/sbin/logrotate /etc/logrotate.conf

Testing Logrotate

You also need to make sure that logrotate will run without any errors, just in case there is a typo in the configuration or it is not configured properly. One way to do it is to wait for the next day for the cron job to be triggered and see that the log files are archived as expected.

A better way is to manually test logrotate so you can catch errors as quickly as possible. There are two ways to do this. The first method is used to simulate the logrotate processing, without actually doing the archiving:

sudo logrotate -d /etc/logrotate.d/app_name

Once this command runs without any errors, it means that the configuration looks good. You can now run the actual archiving process immediately to check the results:

sudo logrotate -f /etc/logrotate.d/app_name

Then proceed to the location of your application’s log files and you should see the original log, and the first archived log. Since the configuration included the delaycompress option, the first archive will not be compressed.

production.log
production.log-20170220

On the following day, the previous day’s archive will now be compressed, so your logs should look something like:

production.log
production.log-20170221
production.log-20170220.gz

 

 

Bonus tip if you are using DelayedJob and Rails: One of the main culprits of a fast-increasing log file is DelayedJob continuously sending debug messages and Rails sending query details into the production log file. To eliminate these kinds of messages in your logs (which sometimes makes debugging harder due to noise), put this in the config/environments/production.rb file:

config.log_level = :info