Is there a way to have an AWS spot instance register with route53 on boot?

Question

We have some test, dev and ci servers that we have setup as long running persistent spot instances mapped to specific domains using route53. This works great - we get the savings, we can allocate these without too much concern about cost but every now and then we loose the instance due to availability. When they come back - they come back with different IP addresses which breaks the route.

Is there a good way to have these instances automatically remap to the new IP address when they come back online (usually within a minute or two)?

Answer 1

Caution: I'm not convinced this approach is working after all. While I can confirm everything runs as I expected it to - the new routes didn't get setup correctly after these machines were assigned new spot instances. I'm not sure if this is because some service is not started by the time this script runs or if Amazon specifically prohibits this behavior. I'd be curious to hear what others have found.

--

N.B. The right answer here might well be using elastic IP addresses which as I understand allow you to have a single static IP address avoiding this issue altogether. I've not done the cost calculation on this but it might well be cheaper than the solution offered below.

What we ended up coming up with is a script that uses the AWS instance metadata and cli to make a route53 call upon reboot. This did NOT work on our old Ubuntu 14.04 instances but appears to on our newer Ubuntu 20.04 instances.

Here's how it works:

1. We built a little script called setupRoute53.sh that knows how to make a single call to route53.
2. We added a job to cron to run this on each reboot.
3. (Bonus) we also created an ansible script to add additional lines to the crontab for each virtual host we're running locally - we reverse proxy multiple services using nginx.

We're currently running this within the ubuntu user - the crontab looks like this:

# m h  dom mon dow   command
@reboot /home/ubuntu/setupRoute53.sh example.com test.example.com

And setupRoute53.sh looks like this:

#!/bin/sh

# setupRoute53.sh

export ROOT_DOMAIN="$1"
export ROUTE53_HOSTNAME="$2"
export IP="$3"

if [ -z "$1" ]; then
    echo "Usage: $1 <route53 domain> <this hostname> ";
    echo "";
    echo "Example: $1 tokenoftrust.com test.tokenoftrust.com";
    echo;
    exit;
fi

if [ -z "$3" ]; then
    echo "IP not given...trying EC2 metadata...";
    IP=$( curl http://169.254.169.254/latest/meta-data/public-ipv4 )
fi

echo "Updating $ROUTE53_HOSTNAME to : $IP"


HOSTED_ZONE_ID=$( aws route53 list-hosted-zones-by-name | grep -B 1 -e "$ROOT_DOMAIN" | sed 's/.*hostedzone\/\([A-Za-z0-9]*\)\".*/\1/' | head -n 1 )
echo "Hosted zone being modified: $HOSTED_ZONE_ID"

INPUT_JSON=$(echo '{
  "Comment": "Update the A record set",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "HOSTNAME",
        "Type": "A",
        "TTL": 60,
        "ResourceRecords": [
          {
            "Value": "127.0.0.1"
          }
        ]
      }
    }
  ]
}' | sed "s/127\.0\.0\.1/$IP/" | sed "s/HOSTNAME/$ROUTE53_HOSTNAME/" )


# http://docs.aws.amazon.com/cli/latest/reference/route53/change-resource-record-sets.html
# We want to use the string variable command so put the file contents (batch-changes file) in the following JSON
INPUT_JSON="{ \"ChangeBatch\": $INPUT_JSON }"

aws route53 change-resource-record-sets --hosted-zone-id "$HOSTED_ZONE_ID" --cli-input-json "$INPUT_JSON"

exit 0;