One day at work I noticed that emails were taking much longer to be sent out from our app. I narrowed down the problem to our background queue which is responsible for sending out the emails. The solution prompted me to write this DelayedJob “survival guide”to help others who may encounter this issue in the future.
In my first job we used Emacs as our main text editor. However, I have no idea how to use it properly so I just used it more like Notepad++ than Emacs. In our team there is one person who refused to use Emacs, instead he was using vi (without the m!). To be honest, it looked painful watching him edit code with no syntax highlighting, no line numbers, and no plugins whatsoever. We tried constantly to convince him to use Emacs but he always refused and stuck to using vi. I thought the guy was crazy.
Fast forward 12 years in my career. I was attending a Ruby conference with my peers and one of the speakers (Brad Urani) demonstrated zshell, vim, and tmux and how he does development in his machine. It looked cool! Some of my peers also uses vim and tmux so its not a new concept to me, however I always decided to use gedit all these years. This time, they probably thought I was crazy.
After the conference I finally decided to jump to the other side. I will no longer use any text editor or IDE for programming and will force myself to use vim exclusively. I also decided to use tmux instead of relying on the guake terminal for more awesomeness.
After two months, am I very pleased with the results: I now have my development environment set up using zshell, vim, and tmux and all the tools that I used to have before have been configured to work on vim. As I feel like I am years (maybe decades) late to the party, here is a simple guide I wrote to help people like me who want to dive head-first into vim and tmux.
This step is not really required to use Vim and Tmux, but it is recommended due to several improvements and plugins that you can use to improve your development environment.
In this guide, I will be using Ubuntu (or other Debian based systems) so I can be lazy and just install it using the package manager:
sudo apt-get install zsh
You can also check the version of the zshell installed to make sure its compatible with the plugins you want to use later on:
Once zshell has been installed, we will want to make it our default shell (instead of let’s say bash). This will make your system use zshell when you invoke the terminal or the command line.
chsh -s $(which zsh)
When you load zshell for the first time, it needs to be configured first and it will prompt you on how to do the initial configuration:
This is the Z Shell configuration function for new users, zsh-newuser-install. You are seeing this message because you have no zsh startup files (the files .zshenv, .zprofile, .zshrc, .zlogin in the directory ~). This function can help you with a few settings that should make your use of the shell easier. You can: (q) Quit and do nothing. The function will be run again next time. (0) Exit, creating the file ~/.zshrc containing just a comment. That will prevent this function being run again. (1) Continue to the main menu. (2) Populate your ~/.zshrc with the configuration recommended by the system administrator and exit (you will need to edit the file by hand, if so desired). --- Type one of the keys in parentheses --- 2
Data breaches are a common occurrence nowadays, with larger and larger personal data being exposed to the public and the hands of malicious organizations. Big companies such as Yahoo, LinkedIn, and Equifax are not immune to these data breaches and resulted in exposing millions of customer accounts. In the Philippines we also experience these data breaches, recently in COL Financial (online stock brokerage) and in Jobstreet (job board). Therefore it is important for you to protect your online accounts to minimize the damage caused by these data breaches.
For quite some time now I had been hesitant to use a password manager. Even though I knew that it is a security best practice, I feel that entrusting all your passwords to one program or organization is scary. This feeling changed recently amid increasing occurrence of data breaches and the amount of data that is being divulged. These events and reflecting how I actually use passwords finally drove me to install and use a password manager.
Password managers are programs that store your passwords in a secure and encrypted format. They also provide tools such as random password generators to make it easier for you to generate secure passwords. These programs can be installed locally in your computer or in the cloud and accessed through the internet.
As your web application grows, there will be times when you need to run scripts or code snippets that could take quite a while to finish. Examples of these are generating large and complex reports, or updating your database with new values. When there are large numbers of records in your application, these scripts may take hours or even days to finish.
We usually access our application using the SSH protocol to log in and perform tasks in the server remotely. This article describes ways to perform long-running scripts or tasks in your application in order from least effective to most effective.
Using an open SSH session
This is the simplest and most direct way to run your script. From the SSH session, just invoke the command directly in the terminal:
bundle exec rake my_long_task:execute
bundle exec rails runner "LongScript.new(arg1, arg2).process"
While the easiest, this is also the most brittle of the methods, as this requires a constant and reliable SSH connection to the remote server. If the SSH client does not receive a response from the server for a set amount of time, it will terminate the connection and kill your script prematurely. Due to this, one workaround is to make your script output messages (via
puts) so that the SSH process does not get terminated. Alternatively you can also consistently send commands to the terminal (by pressing any keys or the Enter key) for this purpose.
It is possible to keep the SSH connection alive without timing out using the
ServerAliveInterval option when you open the SSH connection:
ssh -o ServerAliveInterval=5 -o ServerAliveCountMax=1 $HOST
While these methods work in making sure that the SSH connection is kept alive and does not prematurely terminate, it only solves half of the problem. When your computer (that you use to connect to the remote server) suddenly loses its network connection or loses power due to a battery drain or a power interruption, your long-running script will also be terminated. Due to these potential scenarios the following methods work better than running the scripts directly.
Using a background job library
Background jobs are used by web applications to run code while not blocking the web request-response cycle. A common example for this is sending emails; for instance when you request to reset your password, the application does not need to actually send the password reset email before prompting the user to check his/her email inbox. Usually the code that sends the password reset email is sent to a background job so that the user prompt will be served to the user immediately.
Background jobs are also very helpful in managing error conditions. Usually they have a mechanism of rescuing and handling exceptions and automatically retry the job if it fails for any reason. Using the same example above, if the mail server suddenly stopped working, the background job processor will just retry sending the same email again when the mail server comes back up. Thus the worst case user experience is that the password reset email delivery will be delayed, compare this to the user seeing a
500 Server Error message or an exception trace when the background job is not utilized.
For running long scripts or tasks, background jobs can also be used to handle the processing. This is better than running the script directly as it no longer requires the local computer to have a continuous connection to the remote server. You just send the long script into the job queue, and then you can log out or turn off your machine and the script will be running in the background queue.
While this method will work well for scripts that take only a few minutes to a few hours to process, this is problematic for scripts that take tens of hours or even days to complete. The reason for this is that background job processors have their own timeout mechanism that will automatically terminate the job after a certain time has passed. This is implemented in the libraries to protect them from running looping tasks or hogging server resources. The server’s operating system could also prematurely terminate the background job processes if they exceed a certain threshold of resource usage.
Because of these caveats when using background jobs, the next and last method is the best option so far.
Using a background process
Instead of using a background job to process the long-running script, we can instead do a combination of running the script directly and running it via a background job: running the script and putting it in the background.
The “nohup &” is used when you want to run a process that does not terminate when you log out of a shell. The & command is used to run commands in the background, and combining this with nohup ensures that the command does not get terminated even if the shell has been logged out. The most basic structure is:
nohup script &
script is the command that you want to run. This can be further improved by placing the output of the script in a log file that you can view later to analyze what has happened. The command now looks like:
nohup script > script.out 2>&1 &
script.out is the log file, and the
2>&1 command just makes sure that
stderr is also placed in
stdout so errors can also be viewed in the log file.
An example command that runs a rake task to start Resque workers in the background looks like:
nohup bundle exec rake resque:work QUEUE="*" --trace > rake.out 2>&1 &
If you have a Ruby script that you want to run:
nohup ruby ./myscript.rb > myscript.out 2>&1 &
If you are using Ruby on Rails and you want to run a specific class in your application, you can use the
rails runner command to invoke the method, and then run it in the background:
nohup bundle exec rails runner "LongScript.new(arg1, arg2).process" > longscript.out 2>&1 &  12345