I have a MySQL database inside a private subnet on AWS. I need a MySQL Terraform provider to connect to this using a BASTION SOCKS5 proxy. The Terraform script is running on my PC. From my (Ubuntu) PC command line I can tunnel with SSH to the MySQL database and connect successfully, however when I use Terraform to do the same, I get a timeout after 5 mins with the following error:
Error: failed to connect to MySQL: could not connect to server: socks connect tcp 127.0.0.1:3306->attd-community-mysql-instance.xxxx.eu-west-1.rds.amazonaws.com:3306: unexpected protocol version 74
The Terraform setup is as follows:
Terraform v1.5.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.5.0
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/petoju/mysql v3.0.37
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~>4.5.0"
}
mysql = {
source = "petoju/mysql"
version = ">= 3.0.37"
}
}
required_version = ">= 0.14.9"
}
provider "aws" {
profile = "default"
region = local.region
default_tags {
tags = {
Source = "three-tier-workshop/src/rds-system-users"
}
}
}
provider "mysql" {
endpoint = "${local.rds_hostname}:${local.rds_port}"
username = local.rds_admin_username
password = local.rds_admin_password
proxy = "socks5://127.0.0.1:3306"
}
resource "mysql_database" "web_tier_db" {
name = "webappdb"
}
The SOCKS5 proxy is setup as follows:
resource "aws_instance" "bastion_host" {
instance_type = "t2.micro"
ami = "ami-01dd271720c1ba44f"
subnet_id = local.bastion_subnet_id
associate_public_ip_address = true
key_name = var.bastion_client.key_name
security_groups = [aws_security_group.bastion_host.id]
}
resource "aws_security_group" "bastion_host" {
name = "attd-bastion_host"
description = "Allow bastion access from specific IP"
vpc_id = local.vpc_id
ingress {
description = "Allow SSH access"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["${var.bastion_client.source_ip}/32"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
Note: The var.bastion_client.source_ip is my local IP.
I create the tunnel through the SOCKS5 proxy as follows:
ssh -i "~/.ssh/haden-bastion-test.pem" -f -N -L 3306:attd-community-mysql-instance.xxxx.eu-west-1.rds.amazonaws.com:3306 ubuntu@<SOCKS_HOST_PUBLIC_IP>
When I run the terraform apply command, I get this output:
data.terraform_remote_state.rds_cluster: Reading...
data.terraform_remote_state.network: Reading...
data.terraform_remote_state.rds_cluster: Read complete after 0s
data.terraform_remote_state.network: Read complete after 0s
data.aws_secretsmanager_secret.cluster_admin_user: Reading...
data.aws_secretsmanager_secret.cluster_admin_user: Read complete after 1s [id=arn:aws:secretsmanager:eu-west-1:####:secret:attd-community-mysql-instance/mysql/credentials/admin-7PnVKe]
data.aws_secretsmanager_secret_version.cluster_admin_user: Reading...
data.aws_secretsmanager_secret_version.cluster_admin_user: Read complete after 0s [id=arn:aws:secretsmanager:eu-west-1:####:secret:attd-community-mysql-instance/mysql/credentials/admin-XXXX|AWSCURRENT]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# mysql_database.web_tier_db will be created
+ resource "mysql_database" "web_tier_db" {
+ default_character_set = "utf8mb4"
+ default_collation = "utf8mb4_general_ci"
+ id = (known after apply)
+ name = "webappdb"
}
---
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
mysql_database.web_tier_db: Creating...
mysql_database.web_tier_db: Still creating... [10s elapsed]
mysql_database.web_tier_db: Still creating... [20s elapsed]
mysql_database.web_tier_db: Still creating... [30s elapsed]
...
mysql_database.web_tier_db: Still creating... [4m40s elapsed]
mysql_database.web_tier_db: Still creating... [4m50s elapsed]
╷
│ Error: failed to connect to MySQL: could not connect to server: socks connect tcp 127.0.0.1:3306->attd-community-mysql-instance.XXXX.eu-west-1.rds.amazonaws.com:3306: unexpected protocol version 74
│
│ with mysql_database.web_tier_db,
│ on database.tf line 1, in resource "mysql_database" "web_tier_db":
│ 1: resource "mysql_database" "web_tier_db" {
│
╵
I expected the terraform apply command to finish after about 30-40 sec and to have created the webappdb database.
Can anybody assist with this please?
If I SSH directly to the Proxy server, and the run a netstat while the terraform script is running, i see the following, from which it looks like the connection is being attempted to the MySQL server, but the server is not responding.
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 384/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 652/sshd: /usr/sbin
tcp 0 0 10.1.1.84:22 xxx.xxx.xxx.xxx:53440 ESTABLISHED 1227/sshd: ubuntu [
tcp 0 0 10.1.1.84:36416 10.1.22.148:3306 TIME_WAIT -
tcp 0 0 10.1.1.84:45844 10.1.22.148:3306 TIME_WAIT -
tcp 0 0 10.1.1.84:49796 10.1.22.148:3306 TIME_WAIT -
tcp 0 0 10.1.1.84:56982 10.1.22.148:3306 TIME_WAIT -
tcp 0 0 10.1.1.84:40508 10.1.22.148:3306 TIME_WAIT -
tcp 0 356 10.1.1.84:22 xxx.xxx.xxx.xxx:46458 ESTABLISHED 1334/sshd: ubuntu [
tcp 0 0 10.1.1.84:46734 10.1.22.148:3306 TIME_WAIT -
tcp6 0 0 :::22 :::* LISTEN 652/sshd: /usr/sbin
udp 0 0 127.0.0.1:323 0.0.0.0:* 464/chronyd
udp 0 0 127.0.0.53:53 0.0.0.0:* 384/systemd-resolve
udp 0 0 10.1.1.84:68 0.0.0.0:* 382/systemd-network
udp6 0 0 ::1:323 :::* 464/chronyd
I have tried to connect directly from the terraform script using proxy = "socks5://<SOCK_HOST_PUBLIC_IP:3306" however I had the same results.
I have tried setting the socks host as a environment variable using export all_proxy="socks5://127.0.0.1:3306" however I had the same results.