I've started a new instance (i-d23f3a5c) 2 months ago and it's becoming inaccessible every day.
Looking at /var/log/messages I can see the dhcp client is requesting a lease quite often and at some point the message "dhclient1306: fork: Cannot allocate memory" appears.
From there the server is not accessible anymore and needs to be rebooted (I suppose it has lost it's internal ip).
Aug 19 08:47:19 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 5 (xid=0x7daf4b5c)
Aug 19 08:47:37 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6 (xid=0x7daf4b5c)
Aug 19 08:48:02 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 13 (xid=0x7daf4b5c)
Aug 19 08:48:37 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 21 (xid=0x7daf4b5c)
Aug 19 08:49:24 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 10 (xid=0x7daf4b5c)
Aug 19 08:49:43 ip-172-31-16-179 dhclient[2144]: No DHCPOFFERS received.
Aug 19 08:50:11 ip-172-31-16-179 dhclient[2144]: No working leases in persistent database - sleeping.
Aug 19 08:51:09 ip-172-31-16-179 dhclient[2144]: fork: Cannot allocate memory
Aug 19 08:59:09 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8 (xid=0x322ab770)
Aug 19 08:59:46 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 19 (xid=0x322ab770)
Aug 19 09:00:15 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 17 (xid=0x322ab770)
Aug 19 09:01:14 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 (xid=0x322ab770)
Aug 19 09:01:47 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 7 (xid=0x322ab770)
Aug 19 09:02:26 ip-172-31-16-179 dhclient[2144]: No DHCPOFFERS received.
Aug 19 09:03:12 ip-172-31-16-179 dhclient[2144]: No working leases in persistent database - sleeping.
Aug 19 09:04:17 ip-172-31-16-179 dhclient[2144]: fork: Cannot allocate memory
Aug 19 09:14:10 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 (xid=0x1c2c358)
Aug 19 09:14:55 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6 (xid=0x1c2c358)
Aug 19 09:15:30 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 11 (xid=0x1c2c358)
Aug 19 09:16:22 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8 (xid=0x1c2c358)
Aug 19 09:17:01 ip-172-31-16-179 dhclient[2144]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 16 (xid=0x1c2c358)
Aug 19 09:17:43 ip-172-31-16-179 dhclient[2144]: No DHCPOFFERS received.
And so on until I reboot it from the EC2 console.
What would be the best course of action ?
remove DHCP - is that possible ?
restart dhcp client on regular basis ?
increase the number of possible fork (issue will probably arise at some point)
change the renew time to a longer time ?
fix the dhcp client ?
Thanks,
Ankit