I am trying to connect AWS Glue with Snowflake by using JDBC custom connector and connection. However after I have created the connection and run my job and call the toDF() method to convert dynamic frame to Pyspark Dataframe I get the following error
File "/tmp/Snowflake_Test_Job.py", line 22, in <module>
df1 = Snowflake_New_Connector_node1.toDF()
File "/opt/amazon/lib/python3.7/site-packages/awsglue/dynamicframe.py", line 147, in toDF
return DataFrame(self._jdf.toDF(self.glue_ctx._jvm.PythonUtils.toSeq(scala_options)), self.glue_ctx)
File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
return_value = get_return_value(
File "/opt/amazon/lib/python3.7/site-packages/pyspark/sql/utils.py", line 190, in deco
return f(*a, **kw)
File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o95.toDF.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (179.39.34.49 executor 2): org.apache.spark.util.TaskCompletionListenerException: null
Previous exception in task: Glue ETL Marketplace: JDBC driver failure, driverClassName: net.snowflake.client.jdbc.SnowflakeDriver, error message: No suitable driver found for rendered JDBC URL
The URL I am using is according to the snowflake documentation I am using the following JDBC jar file snowflake-jdbc-3.13.22.jar
Not sure what is going wrong here? If someone has faced a similar problem and found solution to this please share?
I tried creating connection with other JAR files as well however when I try it gives the same error. I tried creating connection from Glue Job code as well but was getting same error. The Ideal result should have been execution of the query on snowflake.