I am new to AWS Glue and I am encountering an issue while trying to use the "google bigquery connector 0.24.2 for aws glue 3.0" in an AWS Glue Job. The goal is to establish a connection between my BigQuery db and S3 (as the data destination). However, when I run the Glue Job, I receive the following error message: "LAUNCH ERROR | Glue ETL Marketplace - failed to download connector. Please refer to the logs for details." I have attached the necessary policies (AWSEC2ContainerRegistryFullAccess and S3FullAccess) and setup the secret as credentials for the database.
Upon investigating the logs, I discovered the following error:
2023-06-09 10:07:05,453 - main - INFO - Glue ETL Marketplace - Start downloading connector jars for connection: data_warehouse
2023-06-09 10:07:05,742 - main - INFO - Glue ETL Marketplace - using region: us-east-1, proxy: 169.254.76.0:8888 and glue endpoint: https://glue.us-east-1.amazonaws.com to get connection: data_warehouse
2023-06-09 10:07:06,012 - main - INFO - Glue ETL Marketplace - Requesting ECR authorization token for registryIds=709825985650 and region_name=us-east-1.
2023-06-09 10:07:06,048 - main - INFO - Glue ETL Marketplace - Calling ECR HTTP API to get manifest of https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/bigquery:0.24.2-glue3.0.
2023-06-09T15:07:06.134+05:00 2023-06-09 10:07:06,134 - main - INFO - Glue ETL Marketplace - Ran into exception in GET request. Retrying...
2023-06-09T15:07:06.205+05:00 2023-06-09 10:07:06,205 - main - INFO - Glue ETL Marketplace - Ran into exception in GET request. Retrying...
2023-06-09T15:07:06.291+05:00 2023-06-09 10:07:06,290 - main - INFO - Glue ETL Marketplace - Ran into exception in GET request. Retrying...
2023-06-09T15:07:06.370+05:00 2023-06-09 10:07:06,369 - main - INFO - Glue ETL Marketplace - Ran into exception in GET request. No more retries
I have already confirmed that I successfully subscribed to the "google bigquery connector 0.24.2 for aws glue 3.0" and established the necessary connections. I would appreciate any guidance or insights on resolving this issue. Thank you in advance for your assistance!