Does the streamWrite Function from pyspark, only run concurenttly and not in parellel?

17 Views Asked by At

So i have 3 streaming dataframes, where i've done some sort of transformation and now trying to write those 3 dataframes to their respective delta tables.

transformed_geo_df.writeStream \
  .format("delta") \
  .outputMode("append") \
  .option("checkpointLocation", "/tmp/kinesis/_checkpoints/") \
  .table("12e255fc4fcd_geo_table")

What i've noticed is that the previous two writestreams aren't invoked, only the most current write stream is performed. Wondering how i can have all three running in parellel.

Further, I'm not sure if this makes a difference but i'm doing it on the databricks platform?

0

There are 0 best solutions below