How to save varchar2 column in oracle using PySpark?

51 Views Asked by At

Since this bugfix I'm unable to save data as varchar, only clob.

The problem is that oracle does not allow index on cblobs. I need to save ROWIDs in varchar2 columns to manage MLOG tables. My save command is:

delta_mlog_df = df.select(F.col("ROW_ID"),F.explode(F.col("TRANSACTIONS")).alias("TRANSACTION_ID"))
delta_mlog_df.write \
    .format("jdbc") \
    .option("driver", "oracle.jdbc.driver.OracleDriver" ) \
    .option("dbtable", ORIGIN_MLOG_HIST_TABLE) \
    .option("url", ORACLE_URL) \
    .option("user", ORACLE_USER) \
    .option("password", ORACLE_PASSWORD) \
    .mode('overwrite') \
    .save()

The data is saved as:

 TRANSACTION_ID  NUMBER
 ROW_ID          CLOB

There is some way to force some column type when I save the data? (like to_sql dtypes for pandas?)

0

There are 0 best solutions below