Working on a cloudformation template which creates instances in public and private subnet instances in public sub needs to call api from private

Question

I have this cloudformation template(https://github.com/devashish234073/cloud-experiments/blob/main/apps/cloudformation-all-public.json) which launches 5 applications in public subnet each communicating with other using private IPs, in this case since all instances are having a public IP address it works fine.

Of the 5 applications I need to move three into private subnet as those are not needed to be exposed to public so I created a private subnet a private security group allowing connection from the public security group on the port in which application runs. I have also attached a NAT gateway to the private subnet for internet connectivity but still I am not able to make it work here is the cloudformation template(https://github.com/devashish234073/cloud-experiments/blob/main/apps/cloudformation-public-private.json) that creates this.

The difference between the two can be seen by doing the following git diff:

git diff 8d9fdfc8588d689ca37a864e28385d818e46cfb1 d27a3e054d9df258753eec2280ca1393178c754a

Kindly please provide suggestions on how to fix the connectivity.

With the first cloudformation template where all instances are in same subnet the route table looks like this:

And this is the network interface attached to the subnet:

With second cloudformation these are the instances created two paving public IP and other 3 don't I am trying to extablish communication from the instances having public IP to the one having private IP.

The route table of the subnet in which public IP is present looks like this:

And the route table of the other subnet is like this:

I can rephrase my problem without stating about the cloudformation as:

In same vpc which is having an internet gateway I have created two subnets one named public-subnet another having named private-subnet

The public-subnet has IP range 172.31.1.0/24 and the private has 172.31.2.0/24

I launched one instance in public subnet gave it a public IP and I launched an instance in private subnet without any public IP.

I connected to public instance did a ping to the private instance using the private IP it did not work:

Both subnet here using the same route table of VPC so here I added to more route

1. from cidr 172.31.1.0/24 to the private instance
2. from cidr 172.31.2.0/24 to the public instance

Here's how my route table looks now:

The enis are these:

Even after doing this I am not able to do a ping using the private IP.

Kindly please help.

I even explicitly associated the route table with both the subnets still it did not work:

In my security group I have also allowed all inbound ICMP still it did not work:

Answer 1

The issues seems to be with the timing of the NAT gateway attachment , seems like NAT gateway was getting attached after the private instances had already created which was causing the userdata to not complete runing. I have added an script in the userdata to wait till internet connection is obtained:

"UserData": {
              "Fn::Base64": {
                  "Fn::Join": [
                      "",
                      [
                          "#!/bin/bash\n",
                          "function check_internet() {\n",
                          "  wget -q --spider http://www.google.com\n",
                          "  if [ $? -eq 0 ]; then\n",
                          "    echo 'Internet is available'\n",
                          "    return 0\n",
                          "  else\n",
                          "    echo 'Internet is not available, retrying in 30 seconds...'\n",
                          "    sleep 30\n",
                          "    return 1\n",
                          "  fi\n",
                          "}\n",
                          "while ! check_internet; do\n",
                          "  continue\n",
                          "done\n",
                          "apt update -y\n",
                          "apt install git -y\n",
                          "apt install nodejs -y\n",
                          "node -v\n",
                          "git clone https://github.com/devashish234073/cloud-experiments\n",
                          "cd cloud-experiments/apps\n",
                          "node db.js PORT=8989&\n"
                      ]
                  ]
              }