Manage your logs using Logrotate

You have finally set up your own server and your application using this guide, and you feel proud that your app is now running in the vast web. A few days later, when you accessed your application’s URL, you are greeted with a Server Not Responding error! In panic, you SSH into your server to see what is going on, and soon find out that you have just run out of disk space.

What happened?

Most likely the reason is that your log files grew larger and larger as your application is running and it started eating through the entire disk. A couple of possible reasons are:

  • There is a background process that runs database queries every set amount of time, and those queries get logged
  • Using a background queue such as DelayedJob that continuously logs debug messages into the log file
  • Malicious scrapers and bots that repeatedly attempt to gain admin access to your application (by using common login paths such as those used in WordPress, etc)
  • Unnecessary logging of events that have large contents (e.g. logging entire objects or entire request data)

To prevent this scenario we need a way to make sure our logs are cleaned up automatically, so we don’t need to manually remove old logs as they have a tendency to be forgotten. One popular tool for this is a program called logrotate.

Logrotate works by making sure that your log files don’t grow in size unchecked. It compresses the log file and labels it with a timestamp so you can go back to older log files, while keeping the current log file contents only within the specified time range. For example, it can compress and archive logs at the end of the day every day so you begin with a fresh log file at the start of each day.

Installation

Let’s start by installing it, here we assume that we are using Ubuntu or any Debian-based system.

sudo apt-get update
sudo apt-get install logrotate

System Configuration

Logrotate’s configuration file can be found in /etc/logrotate.conf. Modify it using your preferred text editor.

sudo nano /etc/logrotate.conf

By default you will see something like this:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
# compress

This is the system or overall setting of logrotate. It is not recommended that we actually modify this file (although you can if you wish) and it is best practice to use specific configuration files for each log file we want to rotate/archive.

Custom Configuration

The custom configuration files are placed in the /etc/logrotate.d directory. We create a separate file for each log type so it is easier to modify and manage. For example, here is a configuration for rotating Nginx logs:

Start by creating the custom configuration file called “nginx”

sudo nano /etc/logrotate.d/nginx

The contents of the file looks like:

/opt/nginx/logs/access.log { daily rotate 30 compress delaycompress missingok notifempty copytruncate } /opt/nginx/logs/error.log { daily rotate 30 compress delaycompress missingok notifempty copytruncate } read more