I currently have a script that takes data from S3 bucket and produces a data frame. I want to send the data frame to snowflake. Currently I am doing it like this
df_write = df \
.write\
.format(SNOWFLAKE_SOURCE_NAME)\
.options(**sfOptions)\
.option("dbtable","test_table")\
.mode("append")\
.save()
The error produced is
An error occurred while calling o72.save.
: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
at net.snowflake.spark.snowflake.Parameters$MergedParameters.<init>(Parameters.scala:294)
at net.snowflake.spark.snowflake.Parameters$.mergeParameters(Parameters.scala:288)
at net.snowflake.spark.snowflake.DefaultSource.createRelation(DefaultSource.scala:97)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
My thought was that this may be a mismatch version error. Currently I'm using hadoop-aws-2.7.3.jar, spark-snowflake_2.12-2.8.1-spark_3.0.jar, snowflake-jdbc-3.12.8.jar