AWS Site-To-Site: able to ping from AWS to on-prem, but from on-prem to AWS not working

Question

I haven't been able to solve this problem for a few days, I've followed millions of tutorials online but I couldn't find anything about it.

I have an EC2 instance that has as private ip: 172.31.27.40. I have only one VPC (the default one, with 3 subnets).

This is my SG:

On prem I have ip address (public): 1.2.3.4. I created a customer-gateway (with on-prem public ip), a virtual-private-gateway (to which I attached the vpc) and the site-to-site connection.

My 2 tunnels are UP , in Static-Routes I added 192.168.0.0/24 (my on prem subnet). I am using the aws-updown.sh script in the ipsec configuration.

My ipsec config:

conn Tunnel1
auto=start
left=%defaultroute
leftid=1.2.3.4
right=(Outside IP address Tunn1)
type=tunnel
leftauth=psk
rightauth=psk
keyexchange=ikev1
ike=aes128-sha1-modp1024
ikelifetime=8h
esp=aes128-sha1-modp1024
lifetime=1h
keyingtries=%forever
leftsubnet=192.168.0.0/24
rightsubnet=172.31.0.0/16
dpddelay=10s
dpdtimeout=30s
dpdaction=restart
## Please note the following line assumes you only have two tunnels in your Strongswan configuration file. This "mark" value must be unique and may need to be changed based on other entries in your configuration file.
mark=499
## Uncomment the following line to utilize the script from the "Automated Tunnel Healhcheck and Failover" section. Ensure that the integer after "-m" matches the "mark" value above, and <VPC CIDR> is replaced with the CIDR of your VPC
## (e.g. 192.168.1.0/24)
leftupdown="/usr/local/sbin/ipsec-notify.sh -ln Tunnel1 -ll *******/30 -lr ******/30 -m 499 -r 172.31.0.0/16"

This is my route table:

From EC2:
[root@ip-***** ec2-user]# ping 192.168.0.58
PING 192.168.0.58 (192.168.0.58) 56(84) bytes of data.
64 bytes from 192.168.0.58: icmp_seq=1 ttl=64 time=7.82 ms
64 bytes from 192.168.0.58: icmp_seq=2 ttl=64 time=7.84 ms
64 bytes from 192.168.0.58: icmp_seq=3 ttl=64 time=7.76 ms
64 bytes from 192.168.0.58: icmp_seq=4 ttl=64 time=10.8 ms

From On prem:
root@****:/home/utente# ping 172.31.27.40
PING 172.31.27.40 (172.31.27.40) 56(84) bytes of data.
From 169.254.**** icmp_seq=1 Destination Host Unreachable
From 169.254.**** icmp_seq=2 Destination Host Unreachable
From 169.254.**** icmp_seq=3 Destination Host Unreachable
From 169.254.**** icmp_seq=4 Destination Host Unreachable

Can you help me?

Answer 1

TL;DR

The AWS updown script creates the routes but does not provide a src.

1. Open you leftupdown script with an editor.
2. Look for the function add_route()

add_route() {
        IFS=',' read -ra route <<< "${TUNNEL_STATIC_ROUTE}"
        for i in "${route[@]}"; do
            ip route add ${i} dev ${TUNNEL_NAME} metric ${TUNNEL_MARK}
        done
        iptables -t mangle -A FORWARD -o ${TUNNEL_NAME} -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
        iptables -t mangle -A INPUT -p esp -s ${TUNNEL_REMOTE_ENDPOINT} -d ${TUNNEL_LOCAL_ENDPOINT} -j MARK --set-xmark ${TUNNEL_MARK}
        ip route flush table 220
}

4. Add the src IP to the route

From:

[...]
ip route add ${i} dev ${TUNNEL_NAME} metric ${TUNNEL_MARK}
[...]

To:

[...]
ip route add ${i} dev ${TUNNEL_NAME} metric ${TUNNEL_MARK} src MY-LOCAL-ON-PREM-IP
[...]

Notice: Obviously, replace MY-LOCAL-ON-PREM-IP with your corresponding local IP 192.168.0.58!

Explanation

I had the exact same problem. I followed the AWS Documentation and created a Site-To-Site VPN in combination with a virtual private gateway. I downloaded the configuration TXT for Strongswan and followed the instruction inside of it. Specifically, I also used the updown script from the AWS configuration TXT.

Here is my on-prem local IP:

3: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 86:00:00:46:85:95 brd ff:ff:ff:ff:ff:ff
    altname enp0s10
    inet 10.0.0.2/32 brd 10.0.0.2 scope global dynamic ens10
       valid_lft 79692sec preferred_lft 79692sec
    inet6 fe80::8400:ff:fe46:8595/64 scope link
       valid_lft forever preferred_lft forever

Here is my ipsec tunnel config:

conn Tunnel1
        type=tunnel
        auto=start
        keyexchange=ikev2
        authby=psk
        leftid=ON-PREM-PUBLIC-IP
        leftsubnet= 10.0.0.0/24 # my on-prem local network
        right=AWS-PUBLIC-IP
        rightsubnet= 192.168.0.0/16 # my AWS VPC
        aggressive=no
        ikelifetime=28800s
        lifetime=3600s
        margintime=270s
        rekey=yes
        rekeyfuzz=100%
        fragmentation=yes
        replay_window=1024
        dpddelay=30s
        dpdtimeout=120s
        dpdaction=restart
        ike=aes128-sha1-modp1024
        esp=aes128-sha1-modp1024
        keyingtries=%forever

        ## Please note the following line assumes you only have two tunnels in your Strongswan configuration file. This "mark" value must be unique and may need to be changed based on other entries in your configuration file.
        mark=100

        ## Uncomment the following line to utilize the script from the "Automated Tunnel Healhcheck and Failover" section. Ensure that the integer after "-m" matches the "mark" value above, and <VPC CIDR> is replaced with the CIDR of yo
ur VPC
        ## (e.g. 192.168.0.0/16)
        leftupdown="/etc/ipsec.d/aws-updown.sh -ln Tunnel1 -ll 169.254.41.94/30 -lr 169.254.41.93/30 -m 100 -r 192.168.0.0/16"

So far so good. But! The AWS updown script (/etc/ipsec.d/aws-updown.sh) is missing a crucial part. You have to specify the src within the add_route() function! Without the src, the ICMP reply can not find the way back to your On-Prem server. (See TL;DR section.)