Get day of week in Pyspark 3 using date format

Question

In my old Spark2.X code, I had a following line

pageviewsDF.groupBy( date_format(col("capturedAt"), "u-E").alias("Day Of Week") ).sum('req')

That will give Day of Week as 1-Mon, 2-Tue etc.

But now in Spark3 I get an error that u-E not recognised and I can use legacy setting as below

spark.conf.set("spark.sql.legacy.timeParserPolicy","LEGACY")

But is there a way to get same output without using legacy setting?

Answer 1

You can use the expr expression to generate the desired format.

import pyspark.sql.functions as F

...
pageviewsDF.groupBy(F.expr('concat(dayofweek(capturedAt), "-", date_format(capturedAt, "E"))').alias("Day Of Week")).sum('req')