SSH and -bash: fork: Cannot allocate memory VPS Ubuntu

Question

I am hosting my Rails app on Ubuntu 12.04 VPS, Nginx + Unicorn, after deployment everything is fine, but few hours later, when I ssh to the VPS I get this message

-bash: fork: Cannot allocate memory

-bash: wait_for: No record of process 4201 

-bash: wait_for: No record of process 4201 

If I run any command, it would just return

-bash: fork: Cannot allocate memory.

Answer 1

Seems you have run out of memory. Many VPS servers are setup with no swap, so when you run out of memory, it will kill things off in a seemingly random manner.

The easiest way to fix it is to get more memory provisioned to your VPS, likely costing more money. The next best way (other than running less stuff and memory optimizing everything running) would be to add a swap partition or swap file.

For a 1GB swap file (as root):

dd if=/dev/zero of=/swapfile bs=1M count=1024
mkswap /swapfile
swapon  /swapfile

Be sure to add it to /etc/fstab too as:

/swapfile none swap defaults 0 0

That will make it come back after reboot.

Answer 2

To get out of this condition without rebooting, you can trigger the OOM killer manually as follows:

echo 1 > /proc/sys/kernel/sysrq
echo f > /proc/sysrq-trigger
echo 0 > /proc/sys/kernel/sysrq

Having done so you can inspect dmesg to find the process responsible for taking all your memory.