As your web application grows, there will be times when you need to run scripts or code snippets that could take quite a while to finish. Examples of these are generating large and complex reports, or updating your database with new values. When there are large numbers of records in your application, these scripts may take hours or even days to finish.

We usually access our application using the SSH protocol to log in and perform tasks in the server remotely. This article describes ways to perform long-running scripts or tasks in your application in order from least effective to most effective.

Using an open SSH session

This is the simplest and most direct way to run your script. From the SSH session, just invoke the command directly in the terminal:

bundle exec rake my_long_task:execute

bundle exec rails runner "LongScript.new(arg1, arg2).process"

While the easiest, this is also the most brittle of the methods, as this requires a constant and reliable SSH connection to the remote server. If the SSH client does not receive a response from the server for a set amount of time, it will terminate the connection and kill your script prematurely. Due to this, one workaround is to make your script output messages (via print or puts) so that the SSH process does not get terminated. Alternatively you can also consistently send commands to the terminal (by pressing any keys or the Enter key) for this purpose.

It is possible to keep the SSH connection alive without timing out using the ServerAliveInterval option when you open the SSH connection:

ssh -o ServerAliveInterval=5 -o ServerAliveCountMax=1 $HOST

While these methods work in making sure that the SSH connection is kept alive and does not prematurely terminate, it only solves half of the problem. When your computer (that you use to connect to the remote server) suddenly loses its network connection or loses power due to a battery drain or a power interruption, your long-running script will also be terminated. Due to these potential scenarios the following methods work better than running the scripts directly.

Using a background job library

Background jobs are used by web applications to run code while not blocking the web request-response cycle. A common example for this is sending emails; for instance when you request to reset your password, the application does not need to actually send the password reset email before prompting the user to check his/her email inbox. Usually the code that sends the password reset email is sent to a background job so that the user prompt will be served to the user immediately.

In Ruby, some of the most popular background job libraries are Sidekiq, Resque, and DelayedJob.

Background jobs are also very helpful in managing error conditions. Usually they have a mechanism of rescuing and handling exceptions and automatically retry the job if it fails for any reason. Using the same example above, if the mail server suddenly stopped working, the background job processor will just retry sending the same email again when the mail server comes back up.  Thus the worst case user experience is that the password reset email delivery will be delayed, compare this to the user seeing a 500 Server Error message or an exception trace when the background job is not utilized.

For running long scripts or tasks, background jobs can also be used to handle the processing. This is better than running the script directly as it no longer requires the local computer to have a continuous connection to the remote server. You just send the long script into the job queue, and then you can log out or turn off your machine and the script will be running in the background queue.

LongScript.new(arg1, arg2).delay.process

While this method will work well for scripts that take only a few minutes to a few hours to process, this is problematic for scripts that take tens of hours or even days to complete. The reason for this is that background job processors have their own timeout mechanism that will automatically terminate the job after a certain time has passed. This is implemented in the libraries to protect them from running looping tasks or hogging server resources. The server’s operating system could also prematurely terminate the background job processes if they exceed a certain threshold of resource usage.

Because of these caveats when using background jobs, the next and last method is the best option so far.

Using a background process

Instead of using a background job to process the long-running script, we can instead do a combination of running the script directly and running it via a background job: running the script and putting it in the background.

The “nohup &” is used when you want to run a process that does not terminate when you log out of a shell. The & command is used to run commands in the background, and combining this with nohup ensures that the command does not get terminated even if the shell has been logged out. The most basic structure is:

nohup script &

Where script is the command that you want to run. This can be further improved by placing the output of the script in a log file that you can view later to analyze what has happened. The command now looks like:

nohup script > script.out 2>&1 &

Where script.out is the log file, and the 2>&1 command just makes sure that stderr is also placed in stdout so errors can also be viewed in the log file.

An example command that runs a rake task to start Resque workers in the background looks like:

nohup bundle exec rake resque:work QUEUE="*" --trace > rake.out 2>&1 &

If you have a Ruby script that you want to run:

nohup ruby ./myscript.rb > myscript.out 2>&1 &

If you are using Ruby on Rails and you want to run a specific class in your application, you can use the rails runner command to invoke the method, and then run it in the background:

nohup bundle exec rails runner "LongScript.new(arg1, arg2).process" > longscript.out 2>&1 &
 [1] 12345

You will notice that when running these commands in the background, it will output the process identifier (PID) of the command that you will need to note if you want to track the background process. If you want to know that your script is still running, you can list the process using its PID:

ps -p 12345

And if for some reason you want to kill the background process you have created, you can also terminate the process using its PID:

kill 12345

Anytime you want to check the log file of the output, you can view the file using this command:

tail -f longscript.out